What Scraping Public Content Taught Me About Markets and “Virality”

If you’ve ever tried to understand what works in a particular niche — what kind of content gets attention, what doesn’t, what a market actually responds to — the usual approach is to scroll. A lot. And scrolling has an obvious problem: you’re relying on whatever the algorithm decides to show you, which is not the same thing as what’s actually performing well across a space.

I got curious about whether this could be approached more systematically — not by scrolling more, but by pulling structured data about a set of posts and looking at it directly. The tool I used for this happened to be a TikTok scraper, but TikTok itself isn’t really the point. It’s one example of a much more general idea: using public data to understand a market, rather than relying on impressions.

What “scraping” actually means

“Scraping” gets thrown around a lot, often without explanation, so it’s worth being precise: it just means pulling publicly visible data from a website or app in a structured way — instead of a person manually copying information, a small program does it, and organizes the result into something usable, like a spreadsheet.

Tools like Apify provide pre-built scrapers for specific platforms — TikTok, Instagram, YouTube, and others — so you don’t need to write the underlying code yourself. Make.com (or similar automation platforms) then connects that scraped data to somewhere useful, like a Google Sheet, and can apply filters automatically. None of this requires programming knowledge; it requires understanding what data you actually want and why.

Why do this at all?

• The honest answer is: to replace impressions with numbers. If you’re trying to understand a niche — what topics get engagement, what formats work, what a “good” result even looks like — looking at a structured dataset of, say, a few hundred posts tells you something that scrolling for an hour doesn’t: actual distributions. How many posts in this space get meaningful engagement versus how many don’t. What separates the two.
• That’s useful for a few different purposes — market research, understanding a space before entering it, or just satisfying the kind of curiosity that wants evidence rather than vibes.

---

“Viral” means something different depending on where you look

This is where TikTok becomes a useful example rather than the main subject. The system I built used filters like “10,000+ views” and “5%+ engagement” as a rough definition of a high-performing post. But those numbers only make sense for TikTok, in certain niches. They don’t transfer directly anywhere else — and in some places, the entire platform doesn’t transfer either. TikTok is banned in India and a number of other countries, which is itself a useful reminder that “what’s viral” is never a global, fixed thing — it’s specific to a platform, a region, and often a niche within that platform.

The same underlying approach — defining engagement ratios and filtering for outliers — works across platforms, but the thresholds change completely depending on the field:

• A fitness or entertainment niche might see videos regularly cross 100,000+ views, where a 5% engagement rate is fairly common — high reach, lower depth.
• A B2B or finance-adjacent niche (on a platform like LinkedIn or YouTube) might see far lower view counts — a few thousand — but with proportionally higher engagement, because the audience is smaller and more specific.
• An educational or how-to niche often shows the opposite pattern from entertainment: lower immediate engagement, but much longer-tail viewing — a post performing “well” might be one that’s still getting views months later, which raw 24-hour engagement numbers wouldn’t capture at all.

The point isn’t that any of these numbers are “correct.” It’s that defining “viral” or “high-performing” requires understanding the specific market you’re looking at — the same dataset interpreted with TikTok-fitness thresholds would make almost everything in a B2B niche look like a failure, when it isn’t.

Where this approach runs into limits

A few things became clear once I had the data in front of me:

It tells you what worked, not why. A scraped dataset can show you that a post got a lot of engagement. It can’t tell you whether that was because of the topic, the timing, the creator’s existing audience, or something unrepeatable. Treating high numbers as a formula to copy is a common mistake — the data shows outcomes, not causes.

It’s biased toward what already succeeded. By definition, you’re only looking at posts that performed well enough to be visible and scrapeable. This is a form of survivorship bias — you don’t see the hundred similar posts that used the same format and got nothing.

It only covers what’s public and accessible. This matters legally as much as practically — scraping public engagement metrics (views, likes, hashtags) is very different from scraping or reusing private data, and the line between “market research” and something that infringes on a platform’s terms of service or someone’s content rights is worth taking seriously, not waving away.

---

If you want to try this yourself

The mechanical setup is fairly simple: a scraping tool (like Apify) pulls structured data based on a search term, an automation platform (like Make.com) filters that data against thresholds you define, and the results land in a spreadsheet for review. I’ve put together the workflow I used as a starting point here: [Download the Make.com Blueprint Here] — though I’d treat the filtering thresholds as the part worth customizing most, since those are the part that actually depends on your niche, not the platform.

---

A few honest questions

Is this legal? Scraping publicly visible data — view counts, hashtags, engagement numbers — is generally treated differently from scraping private data or republishing content without permission. That said, platform terms of service vary and change, so it’s worth checking current policy for whatever platform you’re working with rather than assuming.

Does this work for platforms other than TikTok? Yes — the same logic applies to Instagram, YouTube, LinkedIn, and others. What changes is the engagement thresholds that define “high-performing,” which depend on the platform and niche, not the tool.

Is this useful if you’re not a content creator? Possibly more useful, actually. Understanding what a market responds to — content as a signal of audience interest — is relevant to product, marketing, and research questions well beyond “should I post this.”

---

ALSO SEE: How I Built an AI-Powered Blog Writing Machine (And You Can Too!) – The TrueOnlooker

---

Closing thought

What stuck with me most wasn’t the tool itself — it was how much the definition of “success” changed depending on where you pointed it. The same dataset, filtered with the same logic, would tell a completely different story in a different niche or on a different platform. That’s a small thing, but it’s a useful reminder: numbers feel objective, but the thresholds that turn numbers into “good” or “bad” are choices — and those choices are where the actual thinking happens.