Technical Documentation Now Has a Second Audience: AI Agents

Technical documentation is no longer written exclusively for human readers. AI assistants, coding agents, crawlers, and other machine clients now access the same pages, but they do so in very different ways.

This change creates several practical questions for documentation teams. Should every page have a Markdown version? Is it worth creating an llms.txt file? Which emerging standards deserve attention, and which ones are still too experimental to justify the investment?

There is no universal benchmark showing exactly what percentage of a documentation website’s audience consists of agents. The answer depends on the platform, subject matter, audience, and measurement method. Instead of searching for one industry-wide percentage, teams should examine three separate issues: whether machine traffic is being measured at all, whether agents can access the content, and whether they can find the information they need without unnecessary requests.

Yaroslav Dvorovenko, Head of Ecosystem at Whitechain, a Layer 2 network by WhiteBIT – a crypto exchange that develops its own technology ecosystem and infrastructure for digital assets – shared his view on technical documentation, DevRel, and the growing role of machine readers.

Why Traditional Analytics Cannot See Most AI Agents

Most analytics platforms identify human visitors through JavaScript. When a page opens in a browser, an analytics script runs and sends an event to the tracking system.

Many AI clients never execute that script. As a result, they are not merely counted inaccurately: they may be completely absent from the analytics dashboard.

A joint study by Vercel and MERJ examined more than 500 million GPTBot requests in late 2024. Researchers did not find a single case in which the bot executed JavaScript.

Because this research is now almost two years old, it is reasonable to question whether agent behavior has changed. More recent findings, however, point in much the same direction.

In April 2026, Cisco DevNet engineer Oleksii Borysenko published a preprint analyzing HTTP fingerprints from nine coding agents and six AI assistants on a live documentation portal. The paper has not yet undergone peer review, but its observations are consistent with the earlier research.

Most coding agents retrieve pages through lightweight HTTP tools such as curl, axios, or Go net/http. They request the page source without rendering it as a browser would. Aider and OpenCode are notable exceptions because they use Headless Chromium and can execute JavaScript. Agents operating directly inside a full browser form another separate category.

For documentation teams, the practical consequence is simple: if important content appears only after client-side JavaScript runs, many agents and crawlers will not be able to read it.

This can be tested on any documentation page without relying on industry statistics:

If the phrase is visible in the browser but the command returns zero, the text is being inserted by JavaScript rather than delivered in the initial HTML response. For many machine clients, the content is effectively unavailable.

How Large Is the AI Agent Audience?

Claims that bots already generate more traffic than humans have become common. The numbers may look similar, but they often describe different types of activity.

Some studies compare agent requests with human page views on a documentation platform. Others count all automated HTTP requests across a global network. Another report may exclude crawlers and focus only on deliberate documentation reads.

These figures cannot be treated as interchangeable.

Source	What it measures	Figure	Date
Mintlify	agent web requests compared with human page loads on its documentation platform	66%	July 2026
GitBook	agents’ share of “intentional reads,” excluding crawlers	51.8%	May 2026
Cloudflare Radar	automated requests to HTML content across its network	~57%	June 2026
Imperva / Thales	automated share of all web traffic in 2025	53%	April 2026
Ahrefs	valid llms.txt files that received no requests during one month	97%	June 2026

There is no responsible way to combine these values into one definitive percentage. Their denominators, traffic definitions, and datasets are too different.

Nevertheless, the studies point toward the same broader trend: in some areas of the web, particularly technical documentation, machine traffic is approaching or already exceeding human traffic.

The pace of growth may be more meaningful than the current percentage. Mintlify reported that agent traffic on its platform rose from 15.2% at the beginning of 2026 to 66% in July.

GitBook observed a similar pattern over a longer period. Agents represented less than 10% of documentation reads in January 2025, roughly 41% by December, and the majority of “intentional reads” by May 2026.

The methodologies are not identical, but the direction of the two curves is similar. A single vendor’s metric can be influenced by classification rules or the composition of its customer base. When several independent datasets show comparable growth, however, the underlying trend becomes more convincing.

Commercial incentives should still be kept in mind. Both GitBook and Mintlify sell documentation platforms with AI-related features. Rising agent traffic supports the story behind those products. Their figures should not automatically be dismissed, but they should be evaluated separately from data produced by more neutral infrastructure providers.

Cloudflare is closer to an independent source in this comparison. Its estimate is usually described as approximately 57%, although the precise value varies slightly between publications.

That figure also requires context. It refers to automated HTTP requests for HTML content, not to all internet activity. Once mobile applications, social media, video platforms, and other services are included, human users still generate about two-thirds of overall internet activity.

The definition of an “agent” introduces another complication. The term may include:

• training crawlers collecting large quantities of content;
• search bots checking websites for new or updated pages;
• AI assistants retrieving information for user-facing answers;
• runtime coding agents looking for specific technical instructions.

These clients have different goals and traffic patterns, so treating them as one uniform audience can be misleading.

GitBook’s data illustrates this difference. ChatGPT accounts for approximately 54% of measured agent traffic, while Meta AI contributes about 12%. All coding agents combined represent less than 7%.

The average machine reader is therefore more likely to be an AI assistant than a coding agent. Tools such as Cursor and Claude Code make up a smaller share of overall requests, although they remain an especially important audience for API references and developer documentation.

Agent rankings also change quickly from month to month. A current leader should not be treated as a permanent market standard.

Why More Agent Traffic Does Not Mean Better Documentation

Even perfect traffic measurement would not tell a team whether its documentation works well for agents.

An inefficient agent may produce many requests while trying to solve a single task. It can attempt several nonexistent URLs, open unrelated pages, repeat searches, and only then find the relevant reference.

Every failed attempt appears in the logs as additional traffic. A rising number of requests can therefore indicate that agents are struggling rather than that the documentation has become more popular.

A more capable agent may produce the opposite result. It might retrieve one comprehensive reference page, extract the necessary information, and complete the task without opening anything else.

The Cisco preprint suggests that agents can compress a human browsing journey of several pages into only one or two requests. This makes familiar web analytics metrics much less informative.

Time on page has little meaning for a client that downloads HTML and processes it immediately. Bounce rate does not accurately describe an agent that obtains the answer from its first request. Session depth may reward inefficient navigation rather than successful information retrieval.

Two opposing effects can therefore distort the numbers:

• retries and incorrect URLs inflate request totals;
• compressed sessions reduce the number of recorded visits.

Instead of maximizing traffic, teams should measure wasted work. Useful indicators include the number of 404 errors, repeated requests, irrelevant pages opened before the correct one, and the total path required to reach a valid answer.

What llms.txt Can and Cannot Do

The llms.txt file is placed in the root directory of a website and contains a structured list of important pages. It is intended to give language models and AI agents a clearer map of the site.

During the last two years, llms.txt has often been promoted as a way to increase visibility or citations in AI-generated responses. Public research does not support this claim.

Ahrefs analyzed 137,210 domains in June 2026 and found that approximately 28% had an llms.txt file. The researchers noted that this should be treated as an upper estimate because their sample contained more technical websites than the web as a whole.

The more revealing result concerned actual use: 97% of valid llms.txt files received no requests during May.

Even among the remaining 3%, not all requests came from language models. A considerable share was generated by SEO tools checking whether the file existed. GPTBot and Claude Code were among the recognizable AI clients that did request it.

A separate SE Ranking study examined 300,000 domains in November 2025. It found an adoption rate of roughly 10% and no correlation between llms.txt implementation and citations in AI answers.

The company’s citation prediction model became more accurate when the llms.txt variable was removed, suggesting that the file did not provide useful predictive value.

The difference between the 28% adoption rate reported by Ahrefs and the 10% reported by SE Ranking is not necessarily evidence that one study is wrong. The two samples represent different parts of the web. Ahrefs analyzed a more technically oriented audience, while SE Ranking covered a broader selection of domains.

These results do not make llms.txt pointless. They indicate that its realistic purpose is narrower than some marketing claims suggest.

A training crawler trying to collect large amounts of content may have little reason to use the file. Search bots also appear to place limited importance on it.

A runtime agent behaves differently. If it has already reached a documentation website with a specific task, llms.txt can provide a direct list of relevant routes and prevent the agent from guessing page URLs.

Mintlify tested this scenario in a benchmark involving 2,400 runs across 20 documentation sites with Claude Code and Codex.

Without a sitemap, the agents repeatedly attempted URLs that did not exist and retrieved pages unrelated to the task. Once they were directed to llms.txt, the frequency of errors fell by almost 90%. The agents also required fewer requests and consumed fewer tokens.

This benchmark should be treated carefully because Mintlify conducted the experiment on documentation sites operated by its own customers. Even so, the result demonstrates a plausible and technically understandable use case: llms.txt can improve navigation for agents already visiting a site.

How documentation serves Markdown is another relevant issue.

Mintlify reported that direct requests to .md URLs increased from 25.1% in February to 54.4% in July. During the same period, requests using the Accept: text/markdown header fell from 25.2% to 8.3%.

The trend does not justify removing either method. Direct Markdown routes already serve a meaningful part of machine traffic, while supporting the Markdown accept header may require only a small server configuration change.

Both options can be checked with simple commands:

The evidence therefore supports a limited conclusion. llms.txt can be useful for technical documentation and API references when runtime agents arrive with a specific question. It should not currently be presented as a proven method for gaining more citations in AI-generated answers.

Google Search and Lighthouse Take Different Views of llms.txt

Google published recommendations for generative search optimization in May 2026. In the section addressing common myths, the company explained that special machine-readable files are not required for appearing in Google Search because its search systems do not use them.

At around the same time, Chrome’s Lighthouse auditing tool introduced an Agentic Browsing category. One of its checks looks for the presence of llms.txt.

The two positions may initially appear inconsistent: Google Search says the file is unnecessary, while another Google product checks whether it exists.

The apparent contradiction disappears when the two use cases are separated.

Google Search is discussing indexing and visibility in search results. Lighthouse Agentic Browsing evaluates whether autonomous clients can navigate and interact with a website efficiently. A file can be irrelevant to search rankings while still helping a particular class of agents locate documentation pages.

Both Google’s generative search guidance and Chrome’s technical documentation emphasize that AI agents may interpret websites through the accessibility tree. This gives semantic HTML and accessibility features a much broader role than llms.txt.

Semantic structure can assist screen readers, browsers, search systems, and many types of machine clients. An llms.txt file is more likely to help a narrower group of agents that actively look for it.

The Lighthouse Agentic Browsing category should also be considered experimental. Its audits are available, but their behavior may change without long-term compatibility guarantees. Teams planning to add the category to continuous integration should account for that instability.

How to Check Whether Agents Can Read Your Documentation

Industry research is useful for understanding general trends, but it cannot determine whether one particular documentation site is agent-friendly.

That can be tested directly.

Lighthouse’s Agentic Browsing category offers a reproducible audit and produces a list of technical findings. This is more useful than trying to imagine how a machine client might interpret the page.

Navigation mode has an important limitation: it analyzes one URL rather than the entire documentation website.

Testing only the homepage may therefore create a misleading impression. A more representative review should include several types of pages:

• the documentation homepage;
• a learning or overview section;
• an API reference page;
• a practical tutorial or quick-start.

Differences between pages within the same website often reveal more useful problems than comparisons between unrelated domains. One section may be available in raw HTML while another relies on JavaScript. Navigation may be clear on tutorials but incomplete in API references.

The audit can be run with:

A separate GitBook experiment provides a more behavioral example of documentation testing.

The researchers created a fictional API that could not have appeared in the agents’ training data. They then gave the same coding task to Codex, Claude Code, and Cursor.

Codex opened and read the complete API reference before generating code. It completed all three runs without errors.

Claude Code and Cursor relied on the quick-start page alone. Both initially received a 422 error and then used the error message to correct their implementations.

The purpose of this experiment is not to establish a permanent ranking between agents. Their behavior and capabilities can change rapidly.

The more valuable lesson concerns documentation structure. If an agent begins with the quick-start, can it easily discover the complete reference before making an avoidable mistake? A documentation entry page should make the next relevant source obvious.

Counting the number of pages an agent downloads does not answer this question. The quality of the path matters more than its raw length.

How We Are Adapting Documentation for AI Agents

Semantic structure and ARIA remain the priority. Google and Lighthouse both describe the accessibility tree as an important way for agents to interpret webpages. Documentation should therefore begin with correct HTML structure, meaningful headings, clear landmarks, and properly implemented ARIA.

There is no universal benchmark showing the exact percentage by which semantic improvements increase agent success. However, the work delivers value even if agent behavior changes. It improves screen-reader accessibility, strengthens content structure, and does not depend on one experimental format.

We are keeping llms.txt and direct Markdown routes. Their maintenance cost is relatively low, and the available benchmarks show a realistic benefit for runtime coding agents.

The quick-start must provide a clear path to the full reference. Some agents stop exploring after the first page they find. The initial guide should therefore link directly to any detailed reference needed to complete the task correctly.

Agentic audits should become part of pre-release checks. They provide concrete warnings and failures that can be reviewed before documentation is published.

A documentation MCP server is not a current priority. The ecosystem is still developing, so this should not yet be presented as a mature or essential part of the infrastructure.

WebMCP has been postponed. Building around an experimental interface before its adoption and audit requirements stabilize would create unnecessary maintenance risk. The decision can be reviewed again when production agents begin using the technology or the associated standards become more stable.

The Infrastructure Cost of AI Crawlers

More machine traffic is not automatically beneficial. For documentation platforms, it can also mean higher bandwidth consumption and infrastructure expenses.

Read the Docs described an extreme case in July 2024. One crawler downloaded 73 terabytes of archived HTML in a single month.

After the platform blocked AI crawlers, its daily traffic fell from approximately 800 GB to 200 GB.

The incident does not represent the current state of the entire market, but it demonstrates how expensive uncontrolled crawling can become.

The response to this problem has gradually moved from individual websites to infrastructure providers.

Cloudflare began blocking AI crawlers by default for new domains in July 2025 and introduced its Pay Per Crawl model. In July 2026, the company expanded this approach with Pay Per Use.

The newer model connects payment not only to downloading a webpage but also to using its content as part of an AI-generated answer.

Starting on September 15, 2026, “mixed” crawlers are also expected to be blocked by default on ad-supported pages for new customers, newly added websites, and users of the free plan.

Cloudflare supports its position partly through crawl-to-referral ratios. The exact values vary depending on the reporting period, but the difference between search and AI crawlers remains substantial.

Google produces approximately fourteen crawls for every referral sent to a website. Some AI crawlers may perform tens of thousands of crawls before producing one referral.

Another finding from 2026 shows that more than half of AI crawling on Cloudflare’s network consists of repeatedly requesting pages whose content has not changed.

Before deciding whether to allow, limit, or block agent traffic, documentation teams need at least two internal measurements:

• how many bytes machine clients transfer;
• how much it costs to serve those requests.

Without this information, it is impossible to assess whether agent traffic is economically useful, neutral, or harmful.

Is It Reasonable Not to Optimize for Agents Yet?

There is a legitimate case for avoiding aggressive investment in agent-specific technology in 2026.

Google has stated that llms.txt does not affect search visibility. Multiple studies have found no connection between the file and AI citations. In one analysis, 97% of valid llms.txt files were not requested at all during the measured month.

The Read the Docs example shows that AI crawlers can create substantial costs. WebMCP remains experimental, while Cloudflare continues moving toward stricter controls and monetization models for automated access.

A large share of the statistics about agent growth also comes from companies that sell AI-enabled documentation platforms. Their commercial interest does not automatically make the data unreliable, but it creates another reason to avoid treating every headline figure as an objective market benchmark.

A cautious approach to unstable technology is therefore reasonable. Teams do not need to implement WebMCP before the standard matures. They also face no demonstrated search penalty for operating without llms.txt.

Semantic markup and accessibility should not be placed in the same category as these experimental tools.

Well-structured pages were important long before modern AI agents appeared. They remain valuable even if current machine-readable formats are replaced by new ones. Improving semantics is therefore a low-risk investment that benefits both human and machine readers.

Will AI Agents Replace Human Documentation Readers?

The decline of Stack Overflow appears to offer an obvious parallel, but official documentation follows a different pattern.

Stack Overflow received approximately 200,000 questions per month at its peak in 2014. By December 2025, the monthly total had fallen below 4,000. That was almost 80% lower than a year earlier.

A peer-reviewed study published in Scientific Reports compared activity on Stack Overflow with developer communities on Reddit. Stack Overflow experienced a significant decline, while Reddit did not show an equivalent drop.

One possible explanation is that developer communities offer more than direct answers. Discussion, opinion, professional identity, and social interaction are difficult for language models to replace. Straightforward information exchange is much easier to reproduce.

If the same pattern applied directly to official documentation, human traffic should also be declining rapidly. However, a Read the Docs co-founder reported in early 2025 that the platform had not observed a significant decrease.

The difference may come from the role of primary sources. Developers still need official specifications, current API references, product behavior descriptions, and version-specific instructions.

The 2025 Stack Overflow survey supports this explanation. Technical documentation remained the most widely used learning resource among developers, with 68% of respondents selecting it.

Documentation has not lost its human reader. It has added a second audience whose behavior is harder to observe through standard analytics.

Machine readers require different formats, navigate pages differently, and create a different cost when information is missing or ambiguous. Documentation teams now need to account for both audiences.

Conclusion

1. Determine whether analytics can detect clients that do not execute JavaScript. If all tracking depends on a browser script, a portion of machine traffic will remain invisible.
2. Test several types of documentation pages with an agentic audit rather than evaluating only the homepage.
3. Do not measure machine-readable formats solely through search visibility or citation frequency. For runtime agents, navigation errors and unnecessary requests are more informative.
4. Review the methodology behind every traffic statistic. Similar percentages may describe entirely different datasets and categories of machine activity.
5. Prioritize semantic HTML and correct ARIA if resources allow only one foundational improvement.

There is no universal figure showing exactly how much semantic structure improves the performance of AI agents. Its priority comes from a combination of technical logic and low risk.

Clear semantics simultaneously improve accessibility, machine readability, and the long-term stability of content structure. Their value does not depend on whether llms.txt, WebMCP, or another emerging standard becomes widely adopted.

Technical documentation already has a second reader. However, “AI agent” is an umbrella term covering clients with very different goals – from large-scale crawlers to assistants looking for one precise answer.