mlsanityData to 7 Oct 2026

CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy

arXiv 2412.02210 ↗First posted 3 Dec 2024cs.CVcited by 80

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Large Multimodal Models (LMMs) have demonstrated impressive performance in recognizing document images with natural language instructions. However, it remains unclear to what extent capabilities in literacy with rich structure and fine-grained visual challenges. The current landscape lacks a comprehensive benchmark to effectively measure the literate capabilities of LMMs. Existing benchmarks are often limited by narrow scenarios and specified tasks. To this end, we introduce CC-OCR, a comprehensive benchmark that possesses a diverse range of scenarios, tasks, and challenges. CC-OCR comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction. It includes 39 subsets with 7,058 full annotated images, of which 41% are sourced from real applications, and released for the first time. We evaluate nine prominent LMMs and reveal both the strengths and weaknesses of these models, particularly in text grounding, multi-orientation, and hallucination of repetition. CC-OCR aims to comprehensively evaluate the capabilities of LMMs on OCR-centered tasks, facilitating continued progress in this crucial area.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 50 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

14 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document7 Mar
  2. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models13 May

Other references · 12

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  4. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  5. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  6. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  7. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
  8. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  9. 2017Detecting Curve Text in the Wild: New Dataset and New Solutioncited by 82
  10. 2024DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perceptioncited by 80
  11. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 70
  12. 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 32; 2026: 46.

80 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  3. 2026Qwen3.5-Omni Technical Reportcited by 146
  4. 2025POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversioncited by 34
  5. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  6. 2026STEP3-VL-10B Technical Reportcited by 28
  7. 2026MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scalecited by 26
  8. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25
  9. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 22
  10. 2025Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCRcited by 12
  11. 2025Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Modelscited by 12
  12. 2025MMTABREAL: Real-World Benchmark for Multimodal Table Understandingcited by 9
38 more
  1. 2025CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognitioncited by 9
  2. 2026Youtu-Parsing: Perception, Structuring and Recognition via High-Parallelism Decodingcited by 9
  3. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  4. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  5. 2025Res-Bench: Benchmarking the Robustness of Multimodal Large Language Models to Dynamic Resolution Inputcited by 8
  6. 2026Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Modelcited by 8
  7. 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluationscited by 7
  8. 2025Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?cited by 6
  9. 2025DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLMcited by 6
  10. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMscited by 5
  11. 2025Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learningcited by 4
  12. 2025LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?cited by 4
  13. 2025PreP-OCR: A Complete Pipeline for Document Image Restoration and Enhanced OCR Accuracycited by 4
  14. 2025SCORE: A Semantic Evaluation Framework for Generative Document Parsingcited by 4
  15. 2025ThaiOCRBench: A Task-Diverse Benchmark for Vision-Language Understanding in Thaicited by 4
  16. 2026Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligencecited by 4
  17. 2026UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documentscited by 4
  18. 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptationcited by 3
  19. 2026Why Keep Your Doubts to Yourself? Trading Visual Uncertainties in Multi-Agent Bandit Systemscited by 3
  20. 2025MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Modelscited by 2
  21. 2025Training-Free Multimodal Large Language Model Orchestrationcited by 2
  22. 2025MFE-GAN: Efficient GAN-based Framework for Document Image Enhancement and Binarization with Multi-scale Feature Extractioncited by 2
  23. 2026OmniOCR: Generalist OCR for Ethnic Minority Languagescited by 2
  24. 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluationcited by 2
  25. 2026GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scriptscited by 2
  26. 2026ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formatscited by 2
  27. 2026DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agentscited by 2
  28. 2026StrucTab: A Structured Optimization Framework for Table Parsingcited by 2
  29. 2026MORE: A Multilingual Document Parsing Benchmark and Evaluationcited by 2
  30. 2026TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streamingcited by 2
  31. 2025E-ARMOR: Edge case Assessment and Review of Multilingual Optical Character Recognitioncited by 1
  32. 2025Index-Preserving Lightweight Token Pruning for Efficient Document Understanding in Vision-Language Modelscited by 1
  33. 2025TowerVision: Understanding and Improving Multilinguality in Vision-Language Modelscited by 1
  34. 2025Physical Plausibility Reasoning via HCM-GRPO: Empowering Compact Model for Superior Performancecited by 1
  35. 2025FinCriticalED: A Visual Benchmark for Financial Fact-Level OCRcited by 1
  36. 2026SEA-Vision: A Multilingual Benchmark for Comprehensive Document and Scene Text Understanding in Southeast Asiacited by 1
  37. 2026TDATR: Improving End-to-End Table Recognition via Table Detail-Aware Learning and Cell-Level Visual Alignmentcited by 1
  38. 2026CC-OCR V2: Fine-Grained Attribution of LMM Failures in Real-World Visual Document Understandingcited by 1

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
  2. 2026PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks
  3. 2025Context-Independent OCR with Multimodal LLMs: Effects of Image Resolution and Visual Complexity
  4. 2026OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding
  5. 2025See the Text: From Tokenization to Visual Reading
  6. 2025Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark
  7. 2026ClinOCR-Bench: A Comprehensive Clinical Scanned Document Dataset for Optical Character Recognition Model Evaluation
  8. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation