# CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy

arXiv: https://arxiv.org/abs/2412.02210 · HTML: https://mlsanity.com/paper/2412.02210/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 50 references that are in the map.
- First posted on arXiv 3 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 14 of 14)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding](https://mlsanity.com/paper/2403.12895/) (2024)
- [General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model](https://mlsanity.com/paper/2409.01704/) (2024)
- [TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document](https://mlsanity.com/paper/2403.04473/) (2024)
- [KOSMOS-2.5: A Multimodal Literate Model](https://mlsanity.com/paper/2309.11419/) (2023)
- [Detecting Curve Text in the Wild: New Dataset and New Solution](https://mlsanity.com/paper/1712.02170/) (2017)
- [DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception](https://mlsanity.com/paper/2410.12628/) (2024)
- [TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy](https://mlsanity.com/paper/2406.01326/) (2024)
- [Focus Anywhere for Fine-grained Multi-page Document Understanding](https://mlsanity.com/paper/2405.14295/) (2024)

## Built on by (most cited first, 30 of 81)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Qwen3.5-Omni Technical Report](https://mlsanity.com/paper/2604.15804/) (2026)
- [OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning](https://mlsanity.com/paper/2501.00321/) (2024)
- [POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion](https://mlsanity.com/paper/2509.01215/) (2025)
- [STEP3-VL-10B Technical Report](https://mlsanity.com/paper/2601.09668/) (2026)
- [Shifting AI Efficiency From Model-Centric to Data-Centric Compression](https://mlsanity.com/paper/2505.19147/) (2025)
- [MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale](https://mlsanity.com/paper/2604.04771/) (2026)
- [OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning](https://mlsanity.com/paper/2505.17163/) (2025)
- [OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models](https://mlsanity.com/paper/2502.16161/) (2025)
- [Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCR](https://mlsanity.com/paper/2504.11101/) (2025)
- [Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models](https://mlsanity.com/paper/2506.20168/) (2025)
- [DatBench: Discriminative, Faithful, and Efficient VLM Evaluations](https://mlsanity.com/paper/2601.02316/) (2026)
- [Youtu-Parsing: Perception, Structuring and Recognition via High-Parallelism Decoding](https://mlsanity.com/paper/2601.20430/) (2026)
- [Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model](https://mlsanity.com/paper/2607.24904/) (2026)
- [Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review](https://mlsanity.com/paper/2502.16586/) (2025)
- [MMTABREAL: Real-World Benchmark for Multimodal Table Understanding](https://mlsanity.com/paper/2505.21771/) (2025)
- [CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognition](https://mlsanity.com/paper/2509.19768/) (2025)
- [Res-Bench: Benchmarking the Robustness of Multimodal Large Language Models to Dynamic Resolution Input](https://mlsanity.com/paper/2510.16926/) (2025)
- [A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends](https://mlsanity.com/paper/2507.09861/) (2025)
- [Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?](https://mlsanity.com/paper/2505.12766/) (2025)
- [SO-Bench: A Structural Output Evaluation of Multimodal LLMs](https://mlsanity.com/paper/2511.21750/) (2025)
- [DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLM](https://mlsanity.com/paper/2512.10619/) (2025)
- [PreP-OCR: A Complete Pipeline for Document Image Restoration and Enhanced OCR Accuracy](https://mlsanity.com/paper/2505.20429/) (2025)
- [SCORE: A Semantic Evaluation Framework for Generative Document Parsing](https://mlsanity.com/paper/2509.19345/) (2025)
- [ThaiOCRBench: A Task-Diverse Benchmark for Vision-Language Understanding in Thai](https://mlsanity.com/paper/2511.04479/) (2025)
- [UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents](https://mlsanity.com/paper/2602.07038/) (2026)
- [Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning](https://mlsanity.com/paper/2503.18484/) (2025)
- [LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?](https://mlsanity.com/paper/2505.12307/) (2025)
- [Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligence](https://mlsanity.com/paper/2601.12318/) (2026)
- [Why Keep Your Doubts to Yourself? Trading Visual Uncertainties in Multi-Agent Bandit Systems](https://mlsanity.com/paper/2601.18735/) (2026)

## Abstract (arXiv)

Large Multimodal Models (LMMs) have demonstrated impressive performance in recognizing document images with natural language instructions. However, it remains unclear to what extent capabilities in literacy with rich structure and fine-grained visual challenges. The current landscape lacks a comprehensive benchmark to effectively measure the literate capabilities of LMMs. Existing benchmarks are often limited by narrow scenarios and specified tasks. To this end, we introduce CC-OCR, a comprehensive benchmark that possesses a diverse range of scenarios, tasks, and challenges. CC-OCR comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction. It includes 39 subsets with 7,058 full annotated images, of which 41% are sourced from real applications, and released for the first time. We evaluate nine prominent LMMs and reveal both the strengths and weaknesses of these models, particularly in text grounding, multi-orientation, and hallucination of repetition. CC-OCR aims to comprehensively evaluate the capabilities of LMMs on OCR-centered tasks, facilitating continued progress in this crucial area.