Does llms.txt Actually Work for AI Search Optimization? New Research Says No

There is a new file making the rounds in web development and SEO circles. It is called llms.txt, and if you have spent any time in digital marketing communities lately, you have probably heard it described as essential infrastructure for AI search optimization.

The pitch is seductive. Drop a simple text file on your website, and AI systems like ChatGPT, Claude, Perplexity, and Gemini will suddenly understand your content better. In a world where generative AI is reshaping how people discover information online, who would not want an edge in AI search visibility?

But here is the uncomfortable truth that a recent 90-day study across 10 websites revealed. It does not actually work. At least not yet. And probably not for the reasons you would expect.

What is llms.txt and How Does It Work

The llms.txt file concept is straightforward. Much like an XML sitemap helps search engines understand your site structure, or robots.txt tells crawlers what to index, llms.txt is supposed to provide large language models with a clean markdown-formatted summary of your most important pages.

The theory behind llms.txt goes like this. AI systems waste computational resources and tokens parsing through navigation menus, advertisements, tracking scripts, and JavaScript. Give them a clean document pointing to your best content, and they can serve your information more efficiently to users asking questions.

It sounds reasonable. It looks like legitimate web infrastructure. It feels like progress in the emerging field of generative engine optimization.

The llms.txt Research Study and Its Surprising Results

Search Engine Land recently published original research tracking llms.txt implementation across 10 websites spanning finance, B2B SaaS, ecommerce, insurance, and consumer services. The methodology was simple. Measure AI referral traffic for 90 days before llms.txt implementation, then 90 days after.

The results were stark.

Two sites saw traffic gains of 12.5% and 25% increases in AI-referred visitors. Seven sites saw absolutely nothing change. One site declined by nearly 20%.

But here is where the story gets interesting. Those two apparent success stories were not successes at all. At least not because of the llms.txt file.

Why the llms.txt Success Stories Were Not Actually About llms.txt

The site with 25% AI traffic growth happened to launch a major PR campaign around a banking license during the same period. They also restructured their product pages with extractable comparison tables. They published twelve new FAQ pages optimized for featured snippets. They rebuilt their entire resource center with fresh educational content. And they fixed long-standing technical SEO issues that had been blocking crawlers for months.

When a company gets Bloomberg coverage the same month it launches optimized content and fixes crawl errors, you cannot isolate llms.txt as the growth driver.

The 12.5% growth site had published 27 downloadable AI templates three weeks before implementing llms.txt. These were functional tools that solved real problems for users. Google organic traffic to those templates rose 18% during the same period and kept climbing.

In both cases, llms.txt was a bystander to content strategies and technical improvements that would have worked regardless of the file.

No Major AI Company Actually Uses llms.txt

Here is the elephant in the room that llms.txt advocates rarely address.

No major AI provider has publicly committed to parsing llms.txt files.

Not OpenAI. Not Anthropic. Not Google. Not Meta. Not Perplexity.

Google Search Relations team member John Mueller was characteristically direct about the situation. When you look at server logs, AI services do not even check for the llms.txt file. They simply do not request it.

Google briefly added llms.txt files to their developer documentation sites in December 2024, which seemed like a signal of legitimacy from the company behind the sitemap standard. They removed it within 24 hours.

It turned out to be an accidental sitewide CMS update that content teams had not even known about. The llms.txt files that still exist on some Google properties are not findable by default and exist for internal purposes according to Mueller. Not for AI discovery or generative engine optimization.

Why SEO Professionals Are Attracted to llms.txt

There is something deeply human about our attraction to llms.txt as an AI SEO solution. We are witnessing a fundamental shift in how information gets discovered and consumed online, and that is genuinely unsettling for anyone who has built their career on search optimization.

AI systems are increasingly becoming the intermediary between content and the people who need it. ChatGPT search, Perplexity, Google AI Overviews, and Claude are changing user behavior. The rules of this new game are not written yet.

Into that uncertainty walks llms.txt. It is concrete. It is actionable. It is shaped like the web standards we already know and trust. It offers the comfort of doing something in a moment when the ground is shifting beneath our feet.

But looking like infrastructure is not the same as functioning like infrastructure.

What Actually Improves AI Search Visibility and Referral Traffic

The study's two successful sites offer a more instructive lesson than any file specification. Their AI traffic gains came from proven strategies.

Creating functional content assets. Not content marketing fluff. Actual tools like downloadable templates, comparison tables, and calculators. Things that solve problems people bring to AI systems in the first place.

Structuring content for AI extraction. Interest rates, fees, product specifications, and features organized in formats that large language models can pull directly into answers without interpretation or hallucination.

Fixing technical SEO barriers. If AI crawlers cannot access your content due to JavaScript rendering issues, crawl errors, or robots.txt blocking, no amount of llms.txt documentation helps.

Earning authoritative backlinks and press coverage. Links from major publications, industry citations, and mentions from authoritative sources still influence how AI models assess trustworthiness and expertise.

Matching user search intent. Both successful sites answered specific questions people actually ask. What are the best project management templates. How do these interest rates compare. Content that maps directly to user queries gets surfaced by AI systems.

None of this requires llms.txt. All of it produces measurable results in AI referral traffic.

Should You Implement llms.txt on Your Website

The honest answer is probably yes, but with carefully calibrated expectations about what it will and will not accomplish.

If you are building developer tools where AI coding assistants like GitHub Copilot and Cursor are a primary distribution channel, token efficiency genuinely matters. Your users are already interacting with your documentation through AI agents, and clean markdown reduces friction in those interactions.

For ecommerce sites, B2B SaaS companies, publishers, and most other businesses, treat llms.txt like you would treat an XML sitemap. It is useful infrastructure to have in place. It will not hurt your AI visibility. It takes about an hour to implement properly.

But that hour is almost certainly better spent restructuring a product page with extractable comparison data, publishing a genuinely useful FAQ that answers real customer questions, or fixing a crawl error that has been blocking important content for months.

The Future of AI Search Optimization and Generative Engine Optimization

We are in an awkward transitional period for SEO and digital marketing. The old rules of search engine optimization are eroding as AI reshapes discovery. But the new rules of generative engine optimization have not solidified into industry consensus.

That ambiguity creates anxiety among marketers and SEO professionals. And anxiety makes us susceptible to solutions that feel like control even when the evidence does not support them.

llms.txt is in some ways a symptom of that moment in our industry. It is us reaching for familiar patterns like file specifications, standardized protocols, and documented best practices in a landscape that has not agreed on any of those things yet.

The platforms and formats will keep changing as AI technology evolves. What will not change is the fundamental value exchange that has always driven search visibility.

Create something genuinely useful for your target audience. Make it technically accessible to both humans and machines. Structure it for the way people actually seek information. And earn trust through quality, expertise, and authority.

That is less satisfying than dropping a file in your root directory and calling it AI optimization. But it is also the only strategy that has ever actually worked for sustainable organic growth. Regardless of whether the traffic came from Google, ChatGPT, Perplexity, or whatever AI search platform comes next.

Key Takeaways for AI Search Optimization

1. llms.txt is not currently parsed by any major AI provider including OpenAI, Anthropic, Google, or Meta
2. A 90-day study of 10 websites found no measurable impact from llms.txt implementation alone
3. Sites that saw AI traffic growth achieved it through content quality, technical SEO fixes, and PR coverage
4. Focus optimization efforts on extractable content structures, FAQ pages, and fixing crawl issues
5. Treat llms.txt as low-priority infrastructure rather than a core AI SEO strategy