HWE-Bench finds agents fix 70.7% of real hardware bugs

Hardware engineering has been a thin spot for many LLM agent evaluations, which often stop at small HDL generation tasks. HWE-Bench, submitted to arXiv on April 16, 2026 at 07:19:34 UTC, raises the bar by testing agents on real repository-level hardware bug repair rather than isolated components.

The benchmark contains 417 task instances derived from historical bug-fix pull requests across six major open-source projects. The tasks span Verilog/SystemVerilog and Chisel, covering RISC-V cores, SoCs, and security roots-of-trust. Each task is placed in a fully containerized environment, and the agent has to resolve a real bug report with correctness checked by the project's native simulation and regression flows.

The paper evaluates seven LLMs with four agent frameworks. The best agent resolves 70.7% of tasks overall. That headline is strong, but the split is more informative: performance exceeds 90% on smaller cores and drops below 65% on complex SoC-level projects. In other words, today's agents can often make local hardware fixes when the design is bounded, but they still struggle when the repair depends on broader project structure and interactions across artifacts.

The authors trace failures to three stages: fault localization, hardware-semantic reasoning, and coordination across RTL, configuration, and verification files. That diagnosis matters for EDA teams because hardware bugs are rarely just text-editing problems. A fix has to preserve timing assumptions, module interfaces, verification intent, and project-specific build behavior.

For chip teams, the uncomfortable part is variance. A high average can hide an agent that works on one core family and fails on another. That makes task-suite breadth, container fidelity, and regression quality as important as the model name in any benchmark table.

HWE-Bench is not a claim that agents are ready to own hardware design. It is more useful than that: a reproducible pressure test for where they fail. If future systems improve on the hardest SoC-level cases, hardware-aware agents could become practical assistants for regression triage, open-source core maintenance, and verification-driven repair workflows.