# General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model

arXiv: https://arxiv.org/abs/2409.01704 · HTML: https://mlsanity.com/paper/2409.01704/

## Key facts

- Cited by 180 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 51 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 30 of 42)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [EAST: An Efficient and Accurate Scene Text Detector](https://mlsanity.com/paper/1704.03155/) (2017)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images](https://mlsanity.com/paper/1601.07140/) (2016)
- [mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding](https://mlsanity.com/paper/2403.12895/) (2024)
- [TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document](https://mlsanity.com/paper/2403.04473/) (2024)
- [mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding](https://mlsanity.com/paper/2307.02499/) (2023)
- [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model](https://mlsanity.com/paper/2310.05126/) (2023)
- [ICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboard](https://mlsanity.com/paper/1912.09641/) (2019)
- [ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning](https://mlsanity.com/paper/2402.12185/) (2024)
- [TextBoxes: A Fast Text Detector with a Single Deep Neural Network](https://mlsanity.com/paper/1611.06779/) (2016)
- [Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark](https://mlsanity.com/paper/2202.06767/) (2022)
- [ICDAR2017 Competition on Reading Chinese Text in the Wild (RCTW-17)](https://mlsanity.com/paper/1708.09585/) (2017)
- [Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion](https://mlsanity.com/paper/2202.10304/) (2022)

## Built on by (most cited first, 30 of 180)

- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Kimi-VL Technical Report](https://mlsanity.com/paper/2504.07491/) (2025)
- [MinerU: An Open-Source Solution for Precise Document Content Extraction](https://mlsanity.com/paper/2409.18839/) (2024)
- [DeepSeek-OCR: Contexts Optical Compression](https://mlsanity.com/paper/2510.18234/) (2025)
- [OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning](https://mlsanity.com/paper/2501.00321/) (2024)
- [OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations](https://mlsanity.com/paper/2412.07626/) (2024)
- [PaddleOCR 3.0 Technical Report](https://mlsanity.com/paper/2507.05595/) (2025)
- [olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models](https://mlsanity.com/paper/2502.18443/) (2025)
- [Perception-R1: Pioneering Perception Policy with Reinforcement Learning](https://mlsanity.com/paper/2504.07954/) (2025)
- [X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again](https://mlsanity.com/paper/2507.22058/) (2025)
- [PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model](https://mlsanity.com/paper/2510.14528/) (2025)
- [MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm](https://mlsanity.com/paper/2506.05218/) (2025)
- [CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy](https://mlsanity.com/paper/2412.02210/) (2024)
- [Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting](https://mlsanity.com/paper/2505.14059/) (2025)
- [DeepSeek-OCR 2: Visual Causal Flow](https://mlsanity.com/paper/2601.20552/) (2026)
- [SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion](https://mlsanity.com/paper/2503.11576/) (2025)
- [olmOCR 2: Unit Test Rewards for Document OCR](https://mlsanity.com/paper/2510.19817/) (2025)
- [dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model](https://mlsanity.com/paper/2512.02498/) (2025)
- [Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction](https://mlsanity.com/paper/2410.21169/) (2024)
- [POSTA: A Go-to Framework for Customized Artistic Poster Generation](https://mlsanity.com/paper/2503.14908/) (2025)
- [STEP3-VL-10B Technical Report](https://mlsanity.com/paper/2601.09668/) (2026)
- [MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale](https://mlsanity.com/paper/2604.04771/) (2026)
- [OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.02592/) (2024)
- [PP-DocLayout: A Unified Document Layout Detection Model to Accelerate Large-Scale Data Construction](https://mlsanity.com/paper/2503.17213/) (2025)
- [Progressive Multimodal Reasoning via Active Retrieval](https://mlsanity.com/paper/2412.14835/) (2024)
- [Slow Perception: Let's Perceive Geometric Figures Step-by-step](https://mlsanity.com/paper/2412.20631/) (2024)
- [VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generation](https://mlsanity.com/paper/2502.12782/) (2025)
- [OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models](https://mlsanity.com/paper/2502.16161/) (2025)
- [AgentOCR: Reimagining Agent History via Optical Self-Compression](https://mlsanity.com/paper/2601.04786/) (2026)
- [Dolphin-v2: Universal Document Parsing via Scalable Anchor Prompting](https://mlsanity.com/paper/2602.05384/) (2026)

## Abstract (arXiv)

Traditional OCR systems (OCR-1.0) are increasingly unable to meet people's usage due to the growing demand for intelligent processing of man-made optical characters. In this paper, we collectively refer to all artificial optical signals (e.g., plain texts, math/molecular formulas, tables, charts, sheet music, and even geometric shapes) as "characters" and propose the General OCR Theory along with an excellent model, namely GOT, to promote the arrival of OCR-2.0. The GOT, with 580M parameters, is a unified, elegant, and end-to-end model, consisting of a high-compression encoder and a long-contexts decoder. As an OCR-2.0 model, GOT can handle all the above "characters" under various OCR tasks. On the input side, the model supports commonly used scene- and document-style images in slice and whole-page styles. On the output side, GOT can generate plain or formatted results (markdown/tikz/smiles/kern) via an easy prompt. Besides, the model enjoys interactive OCR features, i.e., region-level recognition guided by coordinates or colors. Furthermore, we also adapt dynamic resolution and multi-page OCR technologies to GOT for better practicality. In experiments, we provide sufficient results to prove the superiority of our model.