Meta's AI Crawlers Dominate the Web But Send Almost No Traffic Back

Meta's AI crawlers generated 9.1 billion requests to websites in the second quarter of 2026 alone, more than any other AI company, while returning almost no visitors to the sites they scraped. New data shows a widening gap between how much AI systems take from the web and how much traffic they give back: Anthropic's crawler fetches roughly 4,580 pages for every single visitor it sends, against just five for Google. For anyone tracking AI search visibility, this is the clearest evidence yet that being crawled and being cited are now two very different things.

What changed?

Security firm DataDome's Q2 2026 AI Traffic Report, drawn from 5 trillion signals analysed daily across more than 400 enterprises, found that its network processed 17.7 billion AI agent requests in the quarter, a 45% increase on Q1 and more than double the pace recorded in the previous report. The growth wasn't gradual. April brought 4.77 billion requests, May surged to 6.29 billion (up 32% in a single month), and June held the highest level yet at 6.60 billion requests, according to DataDome.

The real story is who's behind that surge. Meta has effectively taken over the AI web, with Meta-ExternalAgent growing 74% quarter on quarter and Meta-WebIndexer growing 163%. According to MediaCopilot's analysis of the report, Meta's crawlers logged 9.1 billion requests in Q2 2026 without returning meaningful visitor numbers, so publishers watched AI agent traffic surge while their referral lines stayed near zero.

DataDome vice president Jerome Segura put the shift plainly: "We're entering a new phase of the AI web," he said, describing Meta as moving "from 'scrape once for training' to continuous indexing for real-time AI answers."

OpenAI tells a different story. ChatGPT-User actually fetched pages 6% less often in Q2, yet OpenAI's chatbot still commands 80% to 88% of all AI-driven referral traffic and grew referrals 17% quarter on quarter. Among the smaller players, Claude referrals more than doubled to 876,000, Perplexity grew 37%, and Grok collapsed 74% to just 24,000 visits.

A separate analysis from Nobori puts a number on how lopsided this has become. Anthropic crawls about 4,580 pages for every visitor it returns, while Google crawls five. That ratio, which Nobori calls the "crawl-to-refer ratio", is becoming the sharpest way to see what AI crawling actually costs a website against what it gives back. As Digiday notes, this shift means AI visibility can no longer be judged by referral traffic alone.

Why does this matter for your business?

If your server logs show AI crawlers hitting your site constantly but your analytics show almost no AI referral traffic, you're not imagining it. This is now measurable at scale, and it's happening on top of already-declining click-through rates from AI Overviews and AI Mode.

The practical risk is twofold. First, there's an infrastructure cost: every one of those billions of requests uses your bandwidth and server resources, whether or not it ever leads a real person back to your page. Second, and more important for GEO strategy, crawl volume tells you nothing about visibility. Meta's crawlers might be indexing your content constantly for training or retrieval, but that doesn't mean your brand shows up when someone asks Meta AI a question. Being crawled is a precondition for citation, not a guarantee of it.

This also reframes who you should be optimising for. If OpenAI is converting crawl activity into referral traffic far more efficiently than Meta, and Perplexity is growing its referral share too, the platforms worth prioritising for traffic aren't necessarily the ones generating the most server load. The mix of agents driving traffic has shifted noticeably since earlier reports, and Model Context Protocol (MCP) traffic is now a measurable signal in its own right, indicating an ecosystem still in active exploration. That means the landscape you're optimising for this quarter may look different next quarter.

There's also a governance angle. 54% of DataDome's customers have already adopted agent trust policies, according to the report, recognising that at this scale and pace of change, distinguishing between agents that serve the business and those that simply consume infrastructure is no longer optional. If you haven't audited which bots are hitting your site and why, you're behind a growing number of your peers.

What should you do now?

Start by separating "crawled" from "cited" in how you measure AI visibility. Server log analysis will tell you which bots are visiting and how often; a proper AI visibility check will tell you whether that crawling is translating into your brand actually appearing in answers. If you want to see how your brand shows up across AI answer engines rather than guessing from crawl logs alone, it's worth running a free check with Sited's AI visibility audit.

Second, review your robots.txt and crawler access policies with intent, not defaults. Blanket-blocking every AI crawler risks cutting off retrieval-based citations from platforms like Perplexity and ChatGPT that do send meaningful referral traffic back. Blanket-allowing everything means paying the infrastructure cost of high-volume, low-return crawlers like Meta's without any visibility benefit in return.

Third, prioritise your GEO effort by referral efficiency, not raw crawl volume. Given that OpenAI converts crawling into referrals far more effectively than Meta, content built for ChatGPT citation logic, and increasingly Perplexity, is likely to pay back faster than chasing visibility on platforms that crawl heavily but rarely send anyone your way.

Finally, keep an eye on MCP traffic. It's still small, but its emergence as a measurable signal suggests agentic AI systems are starting to interact with sites in new ways that current analytics tools weren't built to capture.

Frequently asked questions

What is the "crawl-to-refer ratio"?

It's the number of pages an AI crawler fetches from your site divided by the number of visitors its parent platform actually sends back. Anthropic's ratio is roughly 4,580 pages crawled for every visitor returned, compared with five for Google, showing how unevenly AI crawling and referral traffic are distributed across platforms.

Why is Meta generating so much AI traffic?

Meta's crawlers, Meta-ExternalAgent and Meta-WebIndexer, grew sharply in Q2 2026 as the company shifted from one-off training scrapes to continuous indexing for real-time AI answers, according to DataDome. Meta-ExternalAgent grew 74% quarter on quarter and Meta-WebIndexer grew 163%, making Meta the dominant source of AI agent traffic on DataDome's network.

Does this mean I should block AI crawlers from my site?

Not necessarily. Blocking indiscriminately risks losing citations from platforms that do send referral traffic, such as ChatGPT and Perplexity. The smarter move is to distinguish high-value agents from high-volume, low-return ones and set access policies accordingly, something over half of DataDome's enterprise customers have already started doing.

Which AI platform sends the most referral traffic to websites?

Despite crawling less intensively than Meta, OpenAI's ChatGPT sends by far the most referral traffic among AI platforms. It commands 80% to 88% of all AI-driven referral traffic and grew referrals 17% quarter on quarter, while Claude and Perplexity are growing from a much smaller base.

How can I check if my brand is actually being cited by AI, not just crawled?

Server logs only show crawler visits, not whether your brand appears in AI-generated answers. A dedicated AI visibility audit, such as the free one at Sited, checks how your brand is represented across AI search platforms directly, which is the only reliable way to know if crawling is translating into visibility.