HN Spotlight: New arXiv Study Questions Whether AGENTS.md Helps Coding Agents

What appeared on Hacker News

A Hacker News post titled "Evaluating AGENTS.md: are they helpful for coding agents?" drew strong technical attention, reaching 184 points and 146 comments at crawl time. The thread links to arXiv paper 2602.11988, submitted on February 12, 2026, which studies a common workflow in agent-assisted coding: adding repository-level guidance files such as AGENTS.md.

Core question and method

The paper asks whether these context files actually improve real-world completion rates. The authors evaluate coding agents in two complementary settings: standard SWE-bench-style tasks using LLM-generated context files that follow agent-developer recommendations, and a second dataset built from repositories that already include developer-committed context files. This design tests both synthetic and real maintenance environments.

Main finding

The headline result is counterintuitive for many teams currently standardizing AGENTS.md templates. Across multiple coding agents and LLMs, the study reports that context files tended to reduce task success compared with running without repository context. It also reports an inference cost increase above 20%. Behaviorally, the context files did change agent execution patterns: agents explored more files and tests and generally respected explicit instructions. But those added requirements often made tasks harder rather than easier.

Operational implication for engineering teams

The practical takeaway is not "never use AGENTS.md." It is to keep repository instructions minimal, high-signal, and directly tied to constraints that matter for correctness or compliance. Overly broad style mandates and long checklists can increase token usage and distract agents from issue resolution. Teams adopting agent workflows should measure task-level win rate and cost impact for each rule they add, instead of assuming more context is always better.

Sources: Hacker News thread · arXiv paper