Should you enrol in Cloudflare's Pay Per Crawl?

For almost every small site or blog, no. Pay Per Crawl and Pay Per Use are built for publishers with scale and reputation, and the September 2026 default-block change mostly applies to new accounts and ad-supported pages, not existing small sites. The sensible move is to decide deliberately which AI crawlers you allow, rather than switching on monetisation and hoping for revenue that isn't coming yet.

What Pay Per Crawl and Pay Per Use actually are

Pay Per Crawl is a gatekeeper. Cloudflare detects when a request comes from an AI crawler rather than a human visitor, and publishers choose whether that crawler gets in for free, pays to get in, or gets blocked entirely, with the choice set per company if you want it that way (Sherwood News). It launched as a private beta feature of AI Crawl Control, letting site owners require payment every time an AI crawler accesses their content (Cloudflare's changelog).

Pay Per Use is a newer, separate product, also in beta. Instead of charging per crawl, it compensates publishers when an AI company later reports that it actually used the content, with the AI company setting the price and the publisher deciding whether to accept it (Search Engine Journal).

The default-block policy, and what changes on September 15, 2026

In July 2025, Cloudflare became the first infrastructure provider to block AI crawlers from accessing content without permission or compensation by default (Cloudflare's press release). Unlike robots.txt, which crawlers can simply ignore, this block happens at the infrastructure level before a crawler even reaches the site (IDX). Major publishers signed on at launch, including Ziff Davis, The Atlantic, ADWEEK, BuzzFeed and Time (Search Engine Land), and Dotdash Meredith's CEO Neil Vogel called it a way to "limit access to our content to those AI partners willing to engage in fair arrangements" (Transparency Coalition).

The next tightening lands on September 15, 2026, when Cloudflare will block AI training and agent crawlers by default on pages that carry ads. Search crawlers stay allowed, but mixed-use crawlers that blend search, training and agent behaviour, including Googlebot, Bingbot and Applebot, get blocked wherever a site owner has chosen to block training (Novaproxy). This default only applies to domains new to Cloudflare after that date, newly added sites, and free-tier accounts that haven't touched the setting. Existing paying customers keep whatever they've already configured, so this is a new-signup default, not a retroactive one (Crawlora). For more on the mechanics of that deadline, see our piece on Cloudflare's Sept 15 AI crawler deadline.

How pay-per-crawl works mechanically

Cloudflare revived HTTP status code 402, "Payment required." When an AI bot hits a paywalled page, it gets a 402 response instead of a flat rejection, and the AI company can then contact Cloudflare or the site operator to agree a price rather than being shut out entirely (Heise). Cloudflare also rolled out a Content-Signal robots.txt directive with a use= parameter, offering three settings: use=immediate (interact but store and reuse nothing), use=reference (the default, index, excerpt and link back) and use=full (summarise and reproduce) (Crawlora). A year after launch, participation remained thin. Cloudflare had only named one Pay Per Crawl partner, ProRata, the startup behind AI search engine Gist.AI (Yahoo Tech).

Why Cloudflare moved from pay-per-crawl to pay-per-use

Cloudflare's own director explained the shift: pay-per-crawl was the "easiest thing to measure: you know it happens," but pay-per-use is "a better signal of quality," since it compensates publishers when content actually shapes an AI answer rather than just being fetched (Press Gazette). The first named Pay Per Use partners are Ceramic.ai, where publishers get paid when their content appears in search results, and You.com, which lets agents pay on demand for premium content delivered in AI answers for enterprise users (Press Gazette). One analysis frames this as a correction to a real flaw in the old model: a single crawl could feed thousands of AI answers or none at all, and Pay Per Use ties payment to the moment value is actually created (Press and Plugins).

But the same analysis notes publishers still don't set prices in any meaningful sense. Nobody typing a question into ChatGPT is choosing which article to pay for; that decision happens at the company level, and AI companies buy the most data for the least cost. A rate card isn't the same as control, because the AI company still decides whether to pay it (Press and Plugins). For more on Cloudflare's monetisation tooling generally, see Cloudflare lets publishers charge AI bots for access.

The data: crawl volume vs. referral traffic

Cloudflare's own 2025 figures show how lopsided the exchange is. Crawl-to-refer ratios rose through 2024, with Anthropic peaking at 500,000:1 and OpenAI peaking at 3,700:1 (InfoQ). More recent 2026 data shows some improvement but a persistent gap: Anthropic's ratio collapsed roughly 18x but still sits at 5,143:1, and OpenAI improved about 31% to 870:1 (Technology Checker). Compare that to Google, whose crawlers generate roughly 14 page visits for every 1 referral sent back, a ratio Cloudflare itself considers a reasonable exchange (ValueAddVC).

Despite the alarm, AI bots are still a small slice of overall traffic. Across 2025, AI bots excluding Googlebot averaged 4.2% of HTML requests on Cloudflare's network, fluctuating between 2.4% and 6.4% (Search Engine Journal). Googlebot alone accounted for 4.5% of HTML requests, more than all other AI bots combined, and crawled roughly three times as many pages as OpenAI's GPTBot and nearly 200 times more than PerplexityBot (Search Engine Journal).

Who this realistically benefits

One practitioner analysis is blunt: this will only matter to the top 1,000 or so sites in the Majestic Million, the elite destinations with thousands of referring subnets. Outside the top 0.1%, Pay Per Crawl probably won't make you a dime, and it could actively damage your visibility (Bill Hartzer). The risk is specific: bloggers, niche hobby sites and mid-size publishers who enable the feature risk disappearing from search and AI results entirely, because AI companies simply won't pay for them and will skip them without anyone noticing the absence (Bill Hartzer). A separate assessment agrees that AI crawlers will likely only pay for very reputable sites with lots of original, regularly published content, and that larger publishers have far more negotiating leverage, leaving smaller sites at risk of being ignored altogether (Measure Minds Group).

Our view: the economics here favour sites that AI companies already need, not sites hoping to be discovered through a paywall. If you're not already a must-have source, charging for access is more likely to get you skipped than paid.

What a small site or blog owner should actually do

A more measured take for small businesses points out that the September 2026 default only changes behaviour on ad-carrying pages, so a typical small business site without a programmatic ad network won't get auto-blocked by this specific change. But the underlying question, which bots can read your site and on what terms, already applies to every site, ad-supported or not. Given that Cloudflare currently lists only two named Pay Per Use partners, don't expect meaningful revenue this quarter. Treat the switch as worth flipping on deliberately, not as a new income line, and aim for the middle ground: allow crawlers tied to AI search visibility that plausibly send you customers, and block or price the ones that only take (Kindly Creative).

In practice that means auditing which bots currently reach your site, deciding per-crawler whether you want in, out, or priced, and revisiting the decision as Pay Per Use matures. It's also worth separating this infrastructure decision from your actual AI visibility work, since getting cited or recommended in ChatGPT and AI Overviews depends far more on content and authority than on crawler billing, as we cover in why AI engines trust earned media over anything you can buy. If you want a clearer read on where your site stands before making crawler decisions, a free AI visibility scorecard can show what's actually being cited today.

The real limitation here is measurement. Cloudflare's own numbers show crawl volume and referral traffic don't move together, and Pay Per Use still relies on AI companies self-reporting when they used your content, which the evidence so far doesn't fully verify.

Frequently asked questions

Will Pay Per Crawl pay me anything as a small blog?

Almost certainly not yet. Adoption is thin, with Cloudflare naming only a handful of partners across both Pay Per Crawl and Pay Per Use, and practitioner analysis suggests AI companies will only pay for very reputable, high-authority sites with regular original content (Measure Minds Group).

Does the September 2026 block affect my existing site?

Only if you're a new Cloudflare signup, add a new site, or are on a free-tier account that hasn't touched the setting, and even then only on pages carrying ads. Existing paying customers keep their current configuration (Crawlora).

What's the difference between Pay Per Crawl and Pay Per Use?

Pay Per Crawl charges AI companies every time their crawler accesses your content, regardless of whether that content is ever used in an answer. Pay Per Use instead pays publishers when the AI company reports actually using the content, which Cloudflare says is a better signal of quality (Press Gazette).

Could enabling Pay Per Crawl hurt my visibility?

Yes, according to practitioner analysis. If AI companies decide your content isn't worth paying for, they'll simply skip it, which can mean disappearing from AI search results rather than earning revenue (Bill Hartzer).