AI crawler control is moving from advisory metadata to enforceable HTTP behavior. Cloudflare has launched Redirects for AI Training, a paid-plan feature that redirects verified AI training crawlers from deprecated pages to the current canonical URL with a 301 status code.

The problem is not theoretical for documentation teams. Cloudflare says AI bots in its AI Crawler category visited developers.cloudflare.com 4.8M times over the last 30 days, and deprecated content was consumed at the same rate as current content. Banners, noindex metadata, and canonical tags all pointed away from old pages, but training crawlers still fetched the stale text.

The new mechanism combines Cloudflare's verified bot classification with canonical tags already present in HTML. When a request comes from a verified AI Crawler category bot such as GPTBot, ClaudeBot, or Bytespider, Cloudflare reads the page response. If the page has a non-self-referencing canonical tag, Cloudflare returns a 301 Moved Permanently response to that canonical URL before serving the old page. Human visitors, search indexers, and AI assistant traffic are not redirected by this rule.

That status-code approach gives site owners a sharper instrument than asking crawlers to infer intent. Cloudflare notes that search engines understand noindex and canonical tags as part of a mature signal system, but AI training pipelines do not necessarily treat those signals as instructions about model training. Blocking crawlers can also remove useful signal entirely, while redirecting them tells the crawler which version of the content is current.

Cloudflare tested the idea on its own documentation. In March 2026, legacy Workers docs were crawled around 46,000 times by OpenAI, 3,600 times by Anthropic, and 1,700 times by Meta. The company links that pattern to a concrete failure mode: in April 2026, a leading AI assistant answered a Wrangler KV question with the deprecated colon syntax instead of the current command form.

After enabling Redirects for AI Training on developers.cloudflare.com, Cloudflare says 100% of AI training crawler requests to pages with non-self-referencing canonical tags were redirected during the first seven days and were not served deprecated content. The feature does not fix data already ingested, and it does not cover unverified crawlers. Still, it is a meaningful web-governance precedent: sites can now express content freshness to AI training crawlers through the protocol layer, not only through text the crawler may ignore.

#ai-crawlers

Cloudflare turns canonical tags into 301s for AI crawlers