NIST AI 800-3 formalizes benchmark and generalized accuracy for AI evaluations

NIST wants benchmark results to mean something more precise

On February 19, 2026, NIST’s Center for AI Standards and Innovation and Information Technology Laboratory published AI 800-3, a report aimed at improving the statistical validity of AI benchmark evaluations. The core argument is that benchmark scores are being used for increasingly important decisions, but the underlying measurement logic is often underspecified. Evaluators may not state what kind of performance they are estimating, whether the benchmark is meant to stand for a broader population of tasks, or how uncertainty should be calculated. NIST argues that these gaps make benchmark-driven decisions harder to justify.

Two different ideas of accuracy

The report’s first major contribution is a clean separation between benchmark accuracy and generalized accuracy. Benchmark accuracy describes performance on the specific set of questions inside a benchmark. Generalized accuracy describes expected performance across the larger universe of questions that the benchmark is supposed to represent. NIST argues that these are not interchangeable concepts and should not automatically be reported as if they were. That distinction matters because many teams implicitly use benchmark results to make claims about broader model capability without making the statistical assumptions explicit.

Why NIST is pushing GLMMs

NIST also proposes generalized linear mixed models, or GLMMs, as a useful addition to the AI evaluation toolbox. In the report, the agency applies the framework to results from 22 frontier LLMs on GPQA-Diamond, BIG-Bench Hard, and Global-MMLU Lite. According to NIST, GLMMs can estimate latent model capability, surface question difficulty patterns, and quantify uncertainty more efficiently than simpler approaches in many cases. The tradeoff is that GLMMs require more modeling assumptions, but NIST argues that those assumptions can be inspected and therefore can expose weaknesses in benchmark design instead of hiding them.

Why this matters beyond academic statistics

AI 800-3 is not a leaderboard paper and it is not a product endorsement. It is a measurement guidance document aimed at evaluators, developers, procurers, and policymakers who need benchmark evidence they can defend. If its framing takes hold, organizations will have to be more explicit about what an evaluation score actually measures, what the confidence interval represents, and whether a benchmark result can be generalized beyond the tested items. In practical terms, NIST is trying to raise the standard for how frontier model performance is reported before benchmark numbers become even more deeply embedded in buying, deployment, and governance decisions.