mlsanityData to 13 Oct 2026

OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models

arXiv 2305.07895 ↗First posted 13 May 2023cs.CV cs.CLcited by 470

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Large models have recently played a dominant role in natural language processing and multimodal vision-language learning. However, their effectiveness in text-related visual tasks remains relatively unexplored. In this paper, we conducted a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks including Text Recognition, Scene Text-Centric Visual Question Answering (VQA), Document-Oriented VQA, Key Information Extraction (KIE), and Handwritten Mathematical Expression Recognition (HMER). To facilitate the assessment of Optical Character Recognition (OCR) capabilities in Large Multimodal Models, we propose OCRBench, a comprehensive evaluation benchmark. OCRBench contains 29 datasets, making it the most comprehensive OCR evaluation benchmark available. Furthermore, our study reveals both the strengths and weaknesses of these models, particularly in handling multilingual text, handwritten text, non-semantic text, and mathematical expression recognition. Most importantly, the baseline results presented in this study could provide a foundational framework for the conception and assessment of innovative strategies targeted at enhancing zero-shot multimodal techniques. The evaluation pipeline and benchmark are available at https://github.com/Yuliang-Liu/MultimodalOCR.

Key facts. Cited by 470 papers in the map (data through 13 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 110 references that are in the map. First posted on arXiv 13 May 2023.

Builds on

45 of 110 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network22 Aug

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2023Visual Instruction Tuningcited by 7,827
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,703
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,052
  11. 2019Towards VQA Models That Can Readcited by 1,705
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,461
32 more
  1. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,295
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  3. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,029
  4. 2021Florence: A New Foundation Model for Computer Visioncited by 803
  5. 2023Instruction Tuning with GPT-4cited by 681
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  7. 2021InfographicVQAcited by 431
  8. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  9. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  10. 2019Scene Text Visual Question Answeringcited by 327
  11. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  12. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  13. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  14. 2019What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysiscited by 191
  15. 2017Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognitioncited by 161
  16. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 158
  17. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 148
  18. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  19. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  20. 2019ICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboardcited by 127
  21. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  22. 2021StrucTexT: Structured Text Understanding with Multi-Modal Transformerscited by 90
  23. 2020On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answeringcited by 78
  24. 2022Syntax-Aware Network for Handwritten Mathematical Expression Recognitioncited by 68
  25. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  26. 2023Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solutioncited by 54
  27. 2019ICDAR 2019 Competition on Scene Text Visual Question Answeringcited by 46
  28. 2022Toward Understanding WordArt: Corner-Guided Transformer for Scene Text Recognitioncited by 45
  29. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38
  30. 2022When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognitioncited by 35
  31. 2022AttentionHTR: Handwritten Text Recognition Based on Attention Encoder-Decoder Networkscited by 12
  32. 2021Winner Team Mia at TextVQA Challenge 2021: Vision-and-Language Representation Learning with Pre-trained Sequence-to-Sequence Modelcited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 2; 2024: 45; 2025: 190; 2026: 233.

470 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,191
  2. 2025Qwen3-VL Technical Reportcited by 2,663
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,586
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,070
  5. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 566
  6. 2026Kimi K2.5: Visual Agentic Intelligencecited by 472
  7. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 450
  8. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  9. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 237
  10. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 233
  11. 2024An Introduction to Vision-Language Modelingcited by 178
  12. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 177
38 more
  1. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  2. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 171
  3. 2025Thyme: Think Beyond Imagescited by 161
  4. 2026Qwen3.5-Omni Technical Reportcited by 161
  5. 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 145
  6. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 135
  7. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 129
  8. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 121
  9. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 120
  10. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  11. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 113
  12. 2025MiMo-VL Technical Reportcited by 111
  13. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 109
  14. 2025Ovis2.5 Technical Reportcited by 105
  15. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 104
  16. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  17. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  18. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 94
  19. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 93
  20. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 89
  21. 2025Visual Spatial Tuningcited by 86
  22. 2025MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigmcited by 84
  23. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  24. 2024CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacycited by 80
  25. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  26. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  27. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 75
  28. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  29. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  30. 2025Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learningcited by 71
  31. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  32. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 71
  33. 2025Kwai Keye-VL 1.5 Technical Reportcited by 70
  34. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  35. 2025Ovis-U1 Technical Reportcited by 66
  36. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 63
  37. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  38. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 52

50 most cited of 470 All 470 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning
  2. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
  3. 2026Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models
  4. 2026OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios
  5. 2025Benchmarking Vision-Language Models on Optical Character Recognition in Dynamic Video Environments
  6. 2025Visual Text Processing: A Comprehensive Review and Unified Evaluation
  7. 2024OCC-MLLM:Empowering Multimodal Large Language Model For the Understanding of Occluded Objects
  8. 2024Do Current Video LLMs Have Strong OCR Abilities? A Preliminary Study