Scrunch AI vs Sited: Which Fits Cloudflare's New AI Crawler Control?

Sited is the better fit if you're a founder or small marketing team who wants a fast, affordable, one-time read on your AI visibility rather than a recurring monitoring subscription. Scrunch AI is built for brand and agency teams that need continuous tracking and a dashboard to run. The distinction matters more now that Cloudflare has given site owners a new way to control which AI crawlers touch their content.

What changed?

Cloudflare rolled out a "Disallow AI Training" control on 16 September 2026, according to The Neuron. The setting lets site owners stay discoverable in major search engines while telling mixed-use crawlers not to use their content for AI training. It's available directly through Cloudflare's crawler controls, giving publishers a way to separate two things that used to be bundled together: being findable by search and AI answer engines, and being scraped to train the next generation of models.

This lands in the same week as several other AI infrastructure moves covered by The Neuron, including OpenAI disclosing six recent model-misalignment cases, Shopify launching ChatGPT Ads, Databricks rolling out its Astra tooling to roughly 3,500 engineers, and NVIDIA opening new paths for writing GPU kernels in Rust. For anyone managing a brand's AI visibility, though, the Cloudflare change is the one that actually touches your website's plumbing.

It builds on a wider shift in how Google surfaces content. On 27-28 August, Google confirmed that AI Overviews can now expand automatically into a longer, AI Mode-style answer, removing the manual "Show More" button for some queries, while a related change began rolling Google's generative UI out of AI Mode and into AI Overviews too, per Harri Digital. More of the answer now lives on the results page itself, which makes the question of which crawlers can even see your content in the first place more urgent, not less.

Why does this matter for your business?

Until now, most crawler management was blunt. A single user agent might fetch your pages both to index them for search or an AI answer engine, and to hoover them up as training data for a future model. Blocking that agent risked losing legitimate citations in ChatGPT, Perplexity or Google's AI Overviews. Allowing it meant your content could end up training a model with no attribution back to you at all.

Cloudflare's new control unbundles that choice. You can keep the door open for crawlers that generate citations and referral traffic, while shutting it for the ones that only harvest content for training runs. That's a meaningful lever for any business that depends on being named, quoted or linked to when someone asks an AI system a question in its category.

The stakes are real. Seer Interactive's September 2025 study of 25.1 million organic impressions across 42 organisations found that brands cited in AI Overviews earn 35% more organic clicks and 91% more paid clicks than those left out entirely, as reported by Jarred Smith. If a badly configured crawler rule accidentally blocks the wrong bot, that halo effect disappears with it. Getting crawler access right is no longer a technical afterthought; it's a visibility decision with a measurable revenue impact.

The tension between publishers and AI platforms has been building for months. USA Today's CEO has said publishers are "getting close now to the point where we'll block Google as well," and 400 newspapers sued OpenAI and Microsoft over unauthorised content use in June 2026, according to Search Engine Land. Cloudflare's control gives smaller businesses, not just newspapers with legal teams, a practical way to draw their own line.

How does this connect to your AI visibility strategy?

A crawler setting only matters if you know which AI engines are actually citing you and how often. That's where the gap between a monitoring platform and a one-off audit becomes relevant, and it's worth being honest about both sides.

Scrunch AI is a genuinely capable brand AI-visibility platform. It monitors how a brand appears across roughly six AI engines, including ChatGPT, Claude, Gemini and Perplexity, and its Agent Experience Platform (AXP) goes further by serving AI-optimised, machine-readable pages directly to crawlers, a direct answer to exactly the kind of crawler-access question Cloudflare's control raises. For a brand or agency team that wants ongoing competitor benchmarking, citation tracking over time, and enterprise features like SSO and a dedicated go-to-market team, that's a legitimate strength, backed by a Series A and established brand customers.

But that comes at a subscription price built for teams with a monitoring budget, not a founder wanting a quick, clear picture of where they stand. Scrunch AI's Starter plan runs from $250 a month billed annually ($300 month-to-month), and Growth starts at $417 a month billed annually ($500 month-to-month), with Enterprise pricing on top; even the entry tier runs to roughly $3,000 or more a year. There's a 7-day free trial, but after that you're maintaining a dashboard and a plan.

Sited takes the opposite approach on purpose. It's a single $50 payment, not a subscription, and it starts with a free scorecard so you can see where you stand before paying anything at all. There's no sales call and no onboarding sequence. You get a point-in-time audit with prioritised fixes rather than a platform to run continuously, which is what a founder or small marketing team needs when the real question is "what do we fix first," not "how do we monitor this forever." For that reader, Sited answers the same question Scrunch AI answers, at a fraction of the cost and in minutes rather than a subscription cycle.

What should you do now?

Start with your robots.txt and crawler settings, because Cloudflare's new control is only useful if you configure it. Check which bots you're currently allowing and which you're blocking, and make sure you're not accidentally cutting off the AI answer engines that send you qualified traffic.

Then check whether AI systems are citing you at all. Running your own domain through Sited's free audit gives you a fast baseline before deciding whether crawler-level changes, content changes, or both, are the priority.

Finally, revisit this every quarter. Crawler policies, AI Overview formats and model defaults are all moving quickly enough in 2026 that a setting which made sense in the summer may need a second look by the autumn.

Frequently asked questions

What does Cloudflare's "Disallow AI Training" control actually do?

It lets site owners stay discoverable in major search engines while telling mixed-use crawlers not to use their content for AI training. In practice, that means separating indexing and citation access from data harvesting for model training, something that used to be bundled under one crawler permission.

Will blocking AI training crawlers hurt my visibility in ChatGPT or Google AI Overviews?

Not if configured correctly, because the control is designed to keep search and answer-engine crawlers active while restricting only the training-focused bots. The risk is misconfiguration, so it's worth checking your settings rather than assuming a blanket block is safe.

Is Scrunch AI a bad choice for AI visibility?

No, it's a solid option for brand and agency teams that want ongoing monitoring, competitor benchmarking and machine-readable page delivery through its AXP layer, backed by enterprise features and established customers. It's simply built and priced for a different buyer than a founder wanting a quick, affordable audit.

How much does Sited cost compared to Scrunch AI?

Sited is a single $50 payment with a free scorecard to start, while Scrunch AI's plans run from $250 a month billed annually for Starter up to $417 a month billed annually for Growth, plus custom Enterprise pricing. For a one-off, prioritised read on your AI visibility, Sited is materially cheaper and faster to act on.

Do I need to change my crawler settings if I'm a small business?

It's worth checking even if you're small, since AI answer engines increasingly decide what gets recommended before a customer ever reaches your website. A five-minute review of your current crawler rules alongside a visibility audit is a reasonable place to start.