Cloudflare finds agent-ready standards nearly absent across top sites

Cloudflare's April 17 post gives the agent web a hard baseline, and the baseline is thin. In its Agent Readiness write-up, Cloudflare says it scanned the 200,000 most visited domains, filtered out categories where agent interaction is less relevant, and measured whether sites expose the standards that let AI agents discover content, read it efficiently, authenticate, and call capabilities without scraping guesswork.

The numbers are the reason this is worth attention. Cloudflare found that 78% of scanned sites have a robots.txt file, but most are still written for search crawlers rather than AI agents. Only 4% declare AI usage preferences through Content Signals, only 3.9% pass Markdown content negotiation with Accept: text/markdown, and emerging discovery standards such as MCP Server Cards and API Catalogs together appear on fewer than 15 sites in the dataset.

The new Agent Readiness score checks four dimensions: discoverability, content, bot access control, and capabilities. Cloudflare also checks agentic commerce standards such as x402, Universal Commerce Protocol, and Agentic Commerce Protocol, though those do not count toward the current score. The point is practical: an owner can type in a URL, see which checks fail, and get prompts that a coding agent can use to add missing support.

Cloudflare is also exposing the tooling in ways agents can call directly. The site at isitagentready.com publishes a stateless MCP server with a scan_site tool and an Agent Skills index, while Cloudflare's URL Scanner now includes an Agent Readiness tab and API option. That makes the score useful for one-off checks, compliance sweeps, and automated site audits.

The more interesting claim is from Cloudflare's own documentation work. By restructuring its developer docs around Markdown access, focused llms.txt files, and richer metadata, Cloudflare says an agent using Kimi-k2.5 through OpenCode consumed 31% fewer tokens and reached correct answers 66% faster than the average non-refined documentation site. If those gains hold across more tools, agent-friendly web standards stop being cosmetic and become latency, cost, and accuracy infrastructure.