mlsanityData to 7 Oct 2026

OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning

arXiv 2501.00321 ↗First posted 31 Dec 2024cs.CV cs.AIcited by 165

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Scoring the Optical Character Recognition (OCR) capabilities of Large Multimodal Models (LMMs) has witnessed growing interest. Existing benchmarks have highlighted the impressive performance of LMMs in text recognition; however, their abilities in certain challenging tasks, such as text localization, handwritten content extraction, and logical reasoning, remain underexplored. To bridge this gap, we introduce OCRBench v2, a large-scale bilingual text-centric benchmark with currently the most comprehensive set of tasks (4x more tasks than the previous multi-scene benchmark OCRBench), the widest coverage of scenarios (31 diverse scenarios), and thorough evaluation metrics, with 10,000 human-verified question-answering pairs and a high proportion of difficult samples. Moreover, we construct a private test set with 1,500 manually annotated images. The consistent evaluation trends observed across both public and private test sets validate the OCRBench v2's reliability. After carefully benchmarking state-of-the-art LMMs, we find that most LMMs score below 50 (100 in total) and suffer from five-type limitations, including less frequently encountered text recognition, fine-grained perception, layout perception, complex element parsing, and logical reasoning. The project website is at: https://99franklin.github.io/ocrbench_v2/

Key facts. Cited by 165 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 110 of its 150 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

110 of 150 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating24 Dec
  2. 2024CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy3 Dec
  3. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming27 Jun
  4. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding8 Apr
  5. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document7 Mar
  6. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding20 Nov
  7. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding4 Jul
  8. 2021From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network22 Aug

Other references · 102

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
38 more
  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  3. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  4. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  5. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  8. 2016A Diagram Is Worth A Dozen Imagescited by 793
  9. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  10. 2024Yi: Open Foundation Models by 01.AIcited by 706
  11. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  12. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  13. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  14. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  15. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  16. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  17. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  18. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  19. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  20. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  21. 2021InfographicVQAcited by 430
  22. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  23. 2019PubLayNet: largest dataset ever for document layout analysiscited by 349
  24. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  25. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  26. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  27. 2019Scene Text Visual Question Answeringcited by 327
  28. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  29. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  30. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  31. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  32. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  33. 2019Image-based table recognition: data, model, and evaluationcited by 233
  34. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  35. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  36. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  37. 2021Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognitioncited by 194
  38. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192

50 most cited of 102 All 102 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 61; 2026: 104.

165 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025Qwen2.5-Omni Technical Reportcited by 704
  3. 2025MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsingcited by 108
  4. 2025Ovis2.5 Technical Reportcited by 103
  5. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  6. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 95
  7. 2025Visual Agentic Reinforcement Fine-Tuningcited by 53
  8. 2026MiniMax Sparse Attentioncited by 34
  9. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 30
  10. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learningcited by 29
  11. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 28
  12. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
38 more
  1. 2025WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?cited by 26
  2. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25
  3. 2026Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scalecited by 25
  4. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 22
  5. 2025Are Large Vision Language Models Good Game Players?cited by 19
  6. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18
  7. 2025Benchmarking Vision-Language Models on Chinese Ancient Documents: From OCR to Knowledge Reasoningcited by 18
  8. 2025NVIDIA Nemotron Nano V2 VLcited by 15
  9. 2025MonkeyOCR v1.5 Technical Report: Unlocking Robust Document Parsing for Complex Patternscited by 13
  10. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 12
  11. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Visioncited by 12
  12. 2025Multimodal LLMs for OCR, OCR Post-Correction, and Named Entity Recognition in Historical Documentscited by 12
  13. 2025Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Modelscited by 12
  14. 2025Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Modelscited by 10
  15. 2026Imagination Helps Visual Reasoning, But Not Yet in Latent Spacecited by 10
  16. 2025CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognitioncited by 9
  17. 2025MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigmcited by 9
  18. 2026Youtu-Parsing: Perception, Structuring and Recognition via High-Parallelism Decodingcited by 9
  19. 2026ParseBench: A Document Parsing Benchmark for AI Agentscited by 9
  20. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  21. 2025OCRGenBench: A Comprehensive Benchmark for Evaluating OCR Generative Capabilitiescited by 8
  22. 2025VACoT: Rethinking Visual Data Augmentation with VLMscited by 8
  23. 2026VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?cited by 8
  24. 2026PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Taskscited by 8
  25. 2025Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognitioncited by 7
  26. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMscited by 7
  27. 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluationscited by 7
  28. 2026MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decodingcited by 7
  29. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmarkcited by 7
  30. 2025DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLMcited by 6
  31. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  32. 2025SCAN: Semantic Document Layout Analysis for Textual and Visual Retrieval-Augmented Generationcited by 5
  33. 2025RoboEgo System Card: An Omnimodal Model with Native Full Duplexitycited by 5
  34. 2025Hyperphantasia: A Benchmark for Evaluating the Mental Visualization Capabilities of Multimodal LLMscited by 5
  35. 2025Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Modelscited by 5
  36. 2025EcoAlign: An Economically Rational Framework for Efficient LVLM Alignmentcited by 5
  37. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMscited by 5
  38. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5

50 most cited of 165 All 165 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models
  2. 2025Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?
  3. 2026Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models
  4. 2026CC-OCR V2: Fine-Grained Attribution of LMM Failures in Real-World Visual Document Understanding
  5. 2025Evaluating Menu OCR and Translation: A Benchmark for Aligning Human and Automated Evaluations in Large Vision-Language Models
  6. 2025Visual Text Processing: A Comprehensive Review and Unified Evaluation
  7. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
  8. 2026LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR