ParseBench tests OCR agents with 167K rules across real documents
Original: ParseBench is the first document parsing benchmark for AI agents View original →
What the tweet revealed
LlamaIndex described ParseBench as the first benchmark built for document parsing in AI agents and pointed to “167K+ rule-based test cases”. That is material because OCR quality for agents is no longer just about readable text. Agents need table structure, chart values, formatting meaning, and page-grounded evidence that can survive downstream decisions.
The LlamaIndex account often posts framework, LlamaParse, and agent infrastructure updates. The linked blog makes this more than a product note: the dataset, evaluation code, and scientific paper are public through Hugging Face, GitHub, and arXiv. That gives developers a way to test their own parser rather than relying only on vendor-written examples.
The benchmark design
ParseBench contains about 2,000 human-verified enterprise document pages and more than 167,000 dense rule-based tests. It evaluates five dimensions: tables, charts, content faithfulness, semantic formatting, and visual grounding. The documents come from public enterprise sources such as insurance filings, financial reports, government documents, and other real-world formats, rather than only academic PDFs or web pages.
The blog says 14 methods were tested across general-purpose vision-language models, specialized document parsers, and LlamaParse modes. The headline result is that LlamaParse Agentic scored 84.9% overall. The same post says only four providers exceeded 50% on charts, formatting scores ranged from 1.0% for Docling to 85.2% for LlamaParse Agentic, and GPT-5 Mini plus Haiku scored below 8% on visual grounding.
The cost section is also concrete. LlamaIndex reports LlamaParse Agentic at about 1.2 cents per page, while the Cost Effective mode is below 0.4 cents per page. Those numbers make ParseBench useful for procurement and architecture decisions, not just model bragging.
What to watch next is whether independent teams reproduce the rankings and whether the promised leaderboard appears. For regulated document agents, the most important metric may be visual grounding because every extracted number eventually needs an audit trail. Source: LlamaIndex source tweet · ParseBench blog · ParseBench GitHub repo
Related Articles
Long-document OCR is bottlenecked by page chunking and growing KV cache. A widely shared post says Baidu’s Unlimited-OCR uses 3B total parameters, 500M active parameters, and a 32K context window to read 40-page documents in one pass.
Why it matters: enterprise OCR failures break agents long before they show up on academic PDF benchmarks. LlamaIndex says ParseBench evaluates about 2,000 human-verified pages with over 167,000 rules across 14 methods on Kaggle.
AI infrastructure competition is being measured in training throughput, not just chip availability. NVIDIA says Blackwell Ultra reached 1,648 TFLOPs per GPU on DeepSeek-V3 671B, about 3x prior delivered performance.