mlsanityData to 14 Oct 2026

General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model

arXiv 2409.01704 ↗First posted 3 Sep 2024cs.CVcited by 180

Not on the map. Why?

Cited by 54 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Traditional OCR systems (OCR-1.0) are increasingly unable to meet people's usage due to the growing demand for intelligent processing of man-made optical characters. In this paper, we collectively refer to all artificial optical signals (e.g., plain texts, math/molecular formulas, tables, charts, sheet music, and even geometric shapes) as "characters" and propose the General OCR Theory along with an excellent model, namely GOT, to promote the arrival of OCR-2.0. The GOT, with 580M parameters, is a unified, elegant, and end-to-end model, consisting of a high-compression encoder and a long-contexts decoder. As an OCR-2.0 model, GOT can handle all the above "characters" under various OCR tasks. On the input side, the model supports commonly used scene- and document-style images in slice and whole-page styles. On the output side, GOT can generate plain or formatted results (markdown/tikz/smiles/kern) via an easy prompt. Besides, the model enjoys interactive OCR features, i.e., region-level recognition guided by coordinates or colors. Furthermore, we also adapt dynamic resolution and multi-page OCR technologies to GOT for better practicality. In experiments, we provide sufficient results to prove the superiority of our model.

Key facts. Cited by 180 papers in the map (data through 14 Oct 2026). Builds on 42 of its 51 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

42 of 51 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  12. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
30 more
  1. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  2. 2019PlotQA: Reasoning over Scientific Plotscited by 321
  3. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 279
  4. 2017EAST: An Efficient and Accurate Scene Text Detectorcited by 248
  5. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  6. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 236
  7. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  8. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  9. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  10. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 160
  11. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 160
  12. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  13. 2019ICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboardcited by 128
  14. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 128
  15. 2016TextBoxes: A Fast Text Detector with a Single Deep Neural Networkcited by 119
  16. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  17. 2017ICDAR2017 Competition on Reading Chinese Text in the Wild (RCTW-17)cited by 105
  18. 2022Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusioncited by 94
  19. 2016Detecting Text in Natural Image with Connectionist Text Proposal Networkcited by 85
  20. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  21. 2020ContourNet: Taking a Further Step toward Accurate Arbitrary-shaped Scene Text Detectioncited by 69
  22. 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62
  23. 2021Adaptive Boundary Proposal Network for Arbitrary Shape Text Detectioncited by 47
  24. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  25. 2018Multi-Oriented Scene Text Detection via Corner Localization and Region Segmentationcited by 42
  26. 2021PP-OCRv2: Bag of Tricks for Ultra Lightweight OCR Systemcited by 39
  27. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  28. 2019Understanding Optical Music Recognitioncited by 37
  29. 2024Sheet Music Transformer: End-To-End Optical Music Recognition Beyond Monophonic Transcriptioncited by 20
  30. 2019ShopSign: a Diverse Scene Text Dataset of Chinese Shop Signs in Street Viewscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 77; 2026: 88.

180 papers.

Sort:

By area:

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  2. 2025Kimi-VL Technical Reportcited by 388
  3. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 267
  4. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 183
  5. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  6. 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 150
  7. 2025PaddleOCR 3.0 Technical Reportcited by 143
  8. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 125
  9. 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 120
  10. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  11. 2025PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Modelcited by 89
  12. 2025MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigmcited by 87
38 more
  1. 2024CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacycited by 81
  2. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 79
  3. 2026DeepSeek-OCR 2: Visual Causal Flowcited by 74
  4. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 53
  5. 2025olmOCR 2: Unit Test Rewards for Document OCRcited by 47
  6. 2025dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Modelcited by 47
  7. 2024Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extractioncited by 44
  8. 2025POSTA: A Go-to Framework for Customized Artistic Poster Generationcited by 36
  9. 2026STEP3-VL-10B Technical Reportcited by 32
  10. 2026MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scalecited by 27
  11. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 26
  12. 2025PP-DocLayout: A Unified Document Layout Detection Model to Accelerate Large-Scale Data Constructioncited by 26
  13. 2024Progressive Multimodal Reasoning via Active Retrievalcited by 25
  14. 2024Slow Perception: Let's Perceive Geometric Figures Step-by-stepcited by 24
  15. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 23
  16. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 23
  17. 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
  18. 2026Dolphin-v2: Universal Document Parsing via Scalable Anchor Promptingcited by 22
  19. 2026Qianfan-OCR: A Unified End-to-End Model for Document Intelligencecited by 21
  20. 2026PaddleOCR-VL-1.6: Expanding the Frontier of Document Parsing with Under-Optimized Region Refinement and Progressive Post-Trainingcited by 21
  21. 2025AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Academic Paperscited by 20
  22. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18
  23. 2025Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understandingcited by 18
  24. 2025MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrievalcited by 17
  25. 2025Logics-Parsing Technical Reportcited by 17
  26. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 17
  27. 2025Context Cascade Compression: Exploring the Upper Limits of Text Compressioncited by 17
  28. 2024READoc: A Unified Benchmark for Realistic Document Structured Extractioncited by 15
  29. 2025KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understandingcited by 15
  30. 2026TextPecker: Rewarding Structural Anomaly Quantification for Enhancing Visual Text Renderingcited by 15
  31. 2026Multimodal OCR: Parse Anything from Documentscited by 14
  32. 2025Perception in Reflectioncited by 13
  33. 2025UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameterscited by 13
  34. 2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understandingcited by 13
  35. 2025Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCRcited by 12
  36. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  37. 2026OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Modelscited by 12
  38. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11

50 most cited of 180 All 180 (JSON)