ParseBench tests OCR agents with 167K rules across real documents

What the tweet revealed

LlamaIndex described ParseBench as the first benchmark built for document parsing in AI agents and pointed to “167K+ rule-based test cases”. That is material because OCR quality for agents is no longer just about readable text. Agents need table structure, chart values, formatting meaning, and page-grounded evidence that can survive downstream decisions.

The LlamaIndex account often posts framework, LlamaParse, and agent infrastructure updates. The linked blog makes this more than a product note: the dataset, evaluation code, and scientific paper are public through Hugging Face, GitHub, and arXiv. That gives developers a way to test their own parser rather than relying only on vendor-written examples.

The benchmark design

ParseBench contains about 2,000 human-verified enterprise document pages and more than 167,000 dense rule-based tests. It evaluates five dimensions: tables, charts, content faithfulness, semantic formatting, and visual grounding. The documents come from public enterprise sources such as insurance filings, financial reports, government documents, and other real-world formats, rather than only academic PDFs or web pages.

The blog says 14 methods were tested across general-purpose vision-language models, specialized document parsers, and LlamaParse modes. The headline result is that LlamaParse Agentic scored 84.9% overall. The same post says only four providers exceeded 50% on charts, formatting scores ranged from 1.0% for Docling to 85.2% for LlamaParse Agentic, and GPT-5 Mini plus Haiku scored below 8% on visual grounding.

The cost section is also concrete. LlamaIndex reports LlamaParse Agentic at about 1.2 cents per page, while the Cost Effective mode is below 0.4 cents per page. Those numbers make ParseBench useful for procurement and architecture decisions, not just model bragging.

What to watch next is whether independent teams reproduce the rankings and whether the promised leaderboard appears. For regulated document agents, the most important metric may be visual grounding because every extracted number eventually needs an audit trail. Source: LlamaIndex source tweet · ParseBench blog · ParseBench GitHub repo