# Ocean-OCR: Towards General OCR Application via a Vision-Language Model

arXiv: https://arxiv.org/abs/2501.15558 · HTML: https://mlsanity.com/paper/2501.15558/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 67 references that are in the map.
- First posted on arXiv 26 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 27 of 27)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://mlsanity.com/paper/2403.18814/) (2024)
- [COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images](https://mlsanity.com/paper/1601.07140/) (2016)
- [mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding](https://mlsanity.com/paper/2403.12895/) (2024)
- [TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document](https://mlsanity.com/paper/2403.04473/) (2024)
- [mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding](https://mlsanity.com/paper/2307.02499/) (2023)
- [InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD](https://mlsanity.com/paper/2404.06512/) (2024)
- [mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding](https://mlsanity.com/paper/2409.03420/) (2024)
- [MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning](https://mlsanity.com/paper/2409.20566/) (2024)
- [DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception](https://mlsanity.com/paper/2407.08303/) (2024)
- [PP-OCRv2: Bag of Tricks for Ultra Lightweight OCR System](https://mlsanity.com/paper/2109.03144/) (2021)
- [TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens](https://mlsanity.com/paper/2410.05261/) (2024)
- [BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices](https://mlsanity.com/paper/2411.10640/) (2024)
- [POINTS1.5: Building a Vision-Language Model towards Real World Applications](https://mlsanity.com/paper/2412.08443/) (2024)
- [BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline](https://mlsanity.com/paper/2408.15079/) (2024)
- [Incidental Scene Text Understanding: Recent Progresses on ICDAR 2015 Robust Reading Competition Challenge 4](https://mlsanity.com/paper/1511.09207/) (2015)
- [DataSculpt: Crafting Data Landscapes for Long-Context LLMs through Multi-Objective Partitioning](https://mlsanity.com/paper/2409.00997/) (2024)

## Built on by (most cited first, 25 of 25)

- [PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model](https://mlsanity.com/paper/2510.14528/) (2025)
- [Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning](https://mlsanity.com/paper/2510.12712/) (2025)
- [MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale](https://mlsanity.com/paper/2604.04771/) (2026)
- [Dolphin-v2: Universal Document Parsing via Scalable Anchor Prompting](https://mlsanity.com/paper/2602.05384/) (2026)
- [AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Academic Papers](https://mlsanity.com/paper/2504.20115/) (2025)
- [AI Sees Your Location, But With A Bias Toward The Wealthy World](https://mlsanity.com/paper/2502.11163/) (2025)
- [VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models](https://mlsanity.com/paper/2503.07575/) (2025)
- [Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs](https://mlsanity.com/paper/2502.16435/) (2025)
- [UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameters](https://mlsanity.com/paper/2512.21095/) (2025)
- [When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding](https://mlsanity.com/paper/2506.05551/) (2025)
- [Real5-OmniDocBench: A Full-Scale Physical Reconstruction Benchmark for Robust Document Parsing in the Wild](https://mlsanity.com/paper/2603.04205/) (2026)
- [Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models](https://mlsanity.com/paper/2506.12776/) (2025)
- [DianJin-OCR-R1: Enhancing OCR Capabilities via a Reasoning-and-Tool Interleaved Vision-Language Model](https://mlsanity.com/paper/2508.13238/) (2025)
- [Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning](https://mlsanity.com/paper/2506.04559/) (2025)
- [DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translation](https://mlsanity.com/paper/2511.18434/) (2025)
- [DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLM](https://mlsanity.com/paper/2512.10619/) (2025)
- [HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decoding](https://mlsanity.com/paper/2602.12957/) (2026)
- [Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence](https://mlsanity.com/paper/2606.15932/) (2026)
- [RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inference](https://mlsanity.com/paper/2605.00392/) (2026)
- [OmniOCR: Generalist OCR for Ethnic Minority Languages](https://mlsanity.com/paper/2602.21042/) (2026)
- [Geometric Risk Control for Vision-Language Model OCR](https://mlsanity.com/paper/2603.19790/) (2026)
- [Bridging Vision, Language, and Mathematics: Pictographic Character Reconstruction with B'ezier Curves](https://mlsanity.com/paper/2511.00076/) (2025)
- [Evaluating Vision-Language Models as a Zero-Shot Learning Alternative to You Only Look Once and Optical Character Recognition for Nigerian License Plate Recognition](https://mlsanity.com/paper/2607.02025/) (2026)
- [HPD-Parsing: Hierarchical Parallel Document Parsing](https://mlsanity.com/paper/2607.18839/) (2026)
- [SupGRPO: Enhancing GRPO with Matching-based Online SFT for Text Spotting](https://mlsanity.com/paper/2609.07081/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) have shown impressive capabilities across various domains, excelling in processing and understanding information from multiple modalities. Despite the rapid progress made previously, insufficient OCR ability hinders MLLMs from excelling in text-related tasks. In this paper, we present Ocean-OCR, a 3B MLLM with state-of-the-art performance on various OCR scenarios and comparable understanding ability on general tasks. We employ Native Resolution ViT to enable variable resolution input and utilize a substantial collection of high-quality OCR datasets to enhance the model performance. We demonstrate the superiority of Ocean-OCR through comprehensive experiments on open-source OCR benchmarks and across various OCR scenarios. These scenarios encompass document understanding, scene text recognition, and handwritten recognition, highlighting the robust OCR capabilities of Ocean-OCR. Note that Ocean-OCR is the first MLLM to outperform professional OCR models such as TextIn and PaddleOCR.