SWE-CI Pushes Coding-Agent Evaluation From One-Shot Fixes to Long-Horizon Maintenance

Why Hacker News found this paper useful

Benchmarks increasingly decide how people talk about coding agents, but many headline numbers still come from narrow bug-fix setups. SWE-CI drew attention on Hacker News because it asks a harder and more realistic question: can an agent keep a real repository healthy through iterative change, not just land one patch that passes tests once?

What SWE-CI proposes

The arXiv abstract presents SWE-CI as a repository-level benchmark built around the Continuous Integration loop. The paper argues that mature software evolves through requirement changes, repeated implementation attempts, and long-running maintenance work, while static one-shot repair benchmarks miss that dynamic. Instead of grading agents only on immediate functional correctness, SWE-CI evaluates long-term maintainability.

The benchmark contains 100 tasks drawn from real repositories. According to the abstract, each task corresponds on average to 233 days of evolution and 71 consecutive commits. Agents are expected to resolve those tasks through dozens of rounds of analysis and coding iterations, which makes the benchmark materially closer to day-to-day software work than a single failing issue paired with one target fix.

What makes it different from SWE-bench style evaluation

The paper directly positions itself against the limits of static repair paradigms. SWE-bench and related datasets have been valuable because they gave the field a common scoreboard for bug fixing. But they mostly reward short-horizon success: understand one issue, produce one patch, and satisfy the evaluation harness. SWE-CI is trying to capture something else entirely: whether an agent can make changes without degrading the codebase over time as the repository keeps moving.

Why it matters

If this benchmark gains traction, it could change how vendors and research groups report coding-agent progress. A model that looks strong on isolated fixes may perform much worse when it must preserve architecture, pass CI repeatedly, and adapt to long development histories. That is why the Hacker News interest makes sense. The paper is not just offering another dataset; it is arguing that the field needs a different definition of software-engineering competence for agents that claim to work inside real codebases.