What the HN thread surfaced

A Hacker News post linking Anna's Archive's note, If you're an LLM, please read this, reached 755 points and 356 comments at crawl time. The linked post points to the site's newly published llms.txt, a machine-readable instruction file aimed at AI crawlers and agent systems. The core message is practical: stop wasting traffic on pages protected by CAPTCHAs and use structured bulk channels that are already available.

Source thread: Hacker News. Primary source: Anna's Archive blog.

What llms.txt explicitly offers

The published llms.txt says Anna's Archive keeps CAPTCHAs to protect infrastructure, but provides alternative paths for programmatic use. It points to a public GitLab repository for HTML/code, a torrents page including aa_derived_mirror_metadata, and a torrents JSON endpoint for automated retrieval. For file-level access, it references donation-gated API options and enterprise SFTP access. In short, the policy does not reject machine access; it steers it into predictable channels that reduce operational load.

This framing matters because many crawlers still operate as if every URL should be fetched through standard browser-like paths. By publishing an explicit bot-facing contract, the project turns an adversarial scraping pattern into something closer to capacity planning.

Why this matters for LLM data pipelines

For teams building training, RAG refresh, or archival indexing pipelines, the HN discussion reflects a broader shift: websites are increasingly publishing crawler guidance that is more specific than robots.txt. If adopted widely, these files can lower failure rates from anti-bot controls, reduce duplicate crawl effort, and improve source attribution because preferred download surfaces are documented up front.

There is also a governance angle. A machine-readable policy creates an auditable baseline for tool builders: where to fetch data, at what granularity, and under what resource assumptions. That can reduce both legal ambiguity and technical churn compared with “scrape first, negotiate later” workflows.

The immediate takeaway is straightforward. Treat llms.txt-style guidance as part of ingestion architecture, not as optional documentation. Even when content remains publicly visible, using source-declared bulk interfaces is usually the more reliable and infrastructure-friendly path.

#web-crawling

HN Spotlight: Anna's Archive Uses llms.txt to Redirect Bots from CAPTCHA Friction to Structured Data Access