What NIST released

The Center for AI Standards and Innovation (CAISI) at NIST announced a draft document, NIST AI 800-2 Practices for Automated Benchmark Evaluations of Language Models, on January 30, 2026, with an update noted on February 10, 2026. NIST says the public comment window runs through March 31, 2026 as a 60-day comment period.

The stated goal is to strengthen validity, transparency, and reproducibility in AI evaluations. Rather than prescribing a single benchmark, the draft focuses on process quality for automated benchmark evaluation workflows used by model developers, deployers, and third-party evaluators.

Core structure of the draft

Define evaluation objectives and select benchmarks
Implement and run evaluations
Analyze and report results
Standardize terms via a supporting glossary

NIST positions this as voluntary guidance and indicates that more guidance for other evaluation paradigms may follow. The target audience is technical teams, but the guidance is also intended to improve evaluation reporting quality for procurement, implementation, and business decision workflows.

Why it matters now

AI benchmark claims are widely used in product marketing, vendor selection, and governance decisions, but reporting practices remain inconsistent. That inconsistency makes cross-model comparison difficult and raises risk for organizations making high-cost deployment decisions based on partial or non-reproducible evaluation artifacts.

This draft is significant because it shifts attention from isolated benchmark scores toward evaluation discipline. If adopted broadly, it could improve comparability across vendors and reduce ambiguity in procurement and risk review processes, especially for enterprises and public-sector buyers.

What stakeholders should do

Teams that produce or consume model evaluations should review the draft and submit practical feedback before March 31, 2026. NIST explicitly requests input on missing practices, unclear sections, and when automated benchmarks are more or less appropriate than alternative evaluation methods. NIST also notes submitted materials may be subject to public disclosure.

#caisi

NIST Opens Public Comment on Draft AI 800-2 Benchmarking Practices