97% of llms.txt Files Go Unread by AI Crawlers, Data Shows

New server-log research shows that 97% of llms.txt files received zero requests from AI crawlers in May 2026, despite adoption of the format growing 8.8 times over the same period. If your team built one hoping it would boost citations in ChatGPT, Gemini or Perplexity, the data suggests that effort isn't translating into AI search visibility, and your time is better spent elsewhere.

What changed?

A new report has put hard numbers behind a suspicion many in the industry already held. llms.txt, the markdown-based standard some publishers use to guide AI crawlers to their content, is barely being used at all.

According to server-log research covered in Digiday, 97% of llms.txt files got zero crawler requests in May 2026, even as adoption of the format grew 8.8 times over the same period.

Originality.ai CEO Jon Gillham summed up the gap between adoption and actual use, as reported by Digiday. He noted that the growth in adoption "shows that publishers are searching for some control over how AI interacts with their content," but adoption isn't usage, and if 97% of these files are never requested, "this isn't an AI visibility strategy yet."

This lands on top of guidance Google has already given publishers. As reported by Digiday, Google's own generative AI search guidance is that llms.txt is not necessary.

The same reporting points to a broader shift happening underneath the surface. AI agent traffic grew 45% in Q2 2026, per data cited by Digiday. In other words, AI systems are crawling and acting on the web more than ever, just not through the channel many publishers assumed would matter most.

Gillham's own framing captures why this matters. He described current adoption as "a low-cost bet on an agent-driven future which might include llms.txt," as reported by Digiday.

Why does this matter for your business?

If your marketing or dev team added an llms.txt file this year believing it would improve your chances of being cited by ChatGPT, Gemini or Perplexity, this data suggests that box-ticking exercise isn't doing much on its own.

It also reinforces a pattern worth tracking closely: AI platforms decide what to cite using signals that have little to do with the technical files a site publishes for crawlers to find. Structure, freshness, third-party authority and how clearly a page answers a specific question all seem to matter more than whether a site has adopted the latest proposed standard.

This lines up with wider research on what actually drives AI citations. Analysis referenced by Growth Memo has found that content structure and direct-answer framing correlate strongly with whether ChatGPT and similar tools choose to cite a page, a pattern that has nothing to do with llms.txt adoption.

There's a parallel worth drawing with location-based businesses. A separate 2026 local visibility index from SOCi found that just 1.2% of locations were recommended by ChatGPT, 11% by Gemini and 7.4% by Perplexity, compared with brands appearing in Google's local 3-pack 35.9% of the time, as reported by Search Engine Land. The report concluded that AI visibility is three to 30 times harder to achieve than ranking well in traditional local search.

The common thread across both stories: simply being technically "available" to AI crawlers, whether through a markdown file or an accurate business listing, doesn't guarantee you show up in the answer. Presence is not the same as visibility.

For founders and marketing leaders, this is a useful correction to a narrative that has built up around technical AI-readiness checklists. Publishing a file, adding schema markup or opening your robots.txt to AI bots are all low-cost, low-risk moves. But the data now shows they aren't, by themselves, moving the needle on whether your brand gets mentioned when someone asks an AI assistant a question in your category.

What should you do now?

Stop treating llms.txt as a visibility lever on its own. It costs little to keep in place if you already have one, but don't expect it to change your citation rate, and don't prioritise building one if you haven't.

Redirect that effort towards the things the data consistently ties to actual citations: content that directly answers specific questions, pages that get refreshed regularly, and third-party mentions on sites AI platforms already trust. The research cited by Growth Memo points repeatedly to freshness and clear question-and-answer structure as far stronger predictors of citation than crawler-facing technical files.

Audit where your brand actually shows up today rather than assuming technical compliance equals coverage. This is exactly the gap a free check can expose quickly: running your brand and key category terms through a tool like Sited's free audit at https://sited.online will show you whether AI platforms are citing you at all, before you invest further in fixes aimed at the wrong problem.

Finally, keep watching the agent traffic trend rather than the file-adoption trend. With AI agent traffic growing 45% in Q2 2026, the more useful question for most brands isn't "do I have an llms.txt file" but "can an AI agent actually find, parse and trust my content when it lands on my site directly."

Frequently asked questions

What is an llms.txt file?

It's a markdown-format file, similar in concept to robots.txt, that some publishers use to tell AI crawlers and large language models which parts of a site to prioritise. It has been promoted as a way to help AI systems understand and cite a site's content more easily.

Does having an llms.txt file hurt my AI search visibility?

No, the data doesn't suggest it causes harm. It simply shows that 97% of these files received zero requests in May 2026, meaning most sites that have one aren't seeing crawlers actually read it, so it's unlikely to be helping either.

Should I remove my llms.txt file?

There's no need to. It costs little to keep, and Jon Gillham himself frames current adoption as a low-cost bet on a future where agentic AI tools might use it more. Just don't treat it as a substitute for genuine content and citation work.

What should I focus on instead of llms.txt?

Prioritise clear, direct-answer content structure, regular content refreshes, and building mentions on third-party sites that AI platforms already trust. These factors show up repeatedly in citation studies as the strongest predictors of whether AI systems mention a brand.

How is this connected to AI agent traffic growth?

Even as technical files like llms.txt go unread, AI agent traffic grew 45% in Q2 2026. That means AI systems are engaging with the web more directly through crawling and browsing behaviour, making genuine content quality and site accessibility more important than niche technical standards.