mlsanityData to 7 Oct 2026

TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document

arXiv 2403.04473 ↗First posted 7 Mar 2024cs.CV cs.AIcited by 173

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

We present TextMonkey, a large multimodal model (LMM) tailored for text-centric tasks. Our approach introduces enhancement across several dimensions: By adopting Shifted Window Attention with zero-initialization, we achieve cross-window connectivity at higher input resolutions and stabilize early training; We hypothesize that images may contain redundant tokens, and by using similarity to filter out significant tokens, we can not only streamline the token length but also enhance the model's performance. Moreover, by expanding our model's capabilities to encompass text spotting and grounding, and incorporating positional information into responses, we enhance interpretability. It also learns to perform screenshot tasks through finetuning. Evaluation on 12 benchmarks shows notable improvements: 5.2% in Scene Text-Centric tasks (including STVQA, TextVQA, and OCRVQA), 6.9% in Document-Oriented tasks (such as DocVQA, InfoVQA, ChartVQA, DeepForm, Kleister Charity, and WikiTableQuestions), and 2.8% in Key Information Extraction tasks (comprising FUNSD, SROIE, and POIE). It outperforms in scene text spotting with a 10.9% increase and sets a new standard on OCRBench, a comprehensive benchmark consisting of 29 OCR-related assessments, with a score of 561, surpassing previous open-sourced large multimodal models for document understanding. Code will be released at https://github.com/Yuliang-Liu/Monkey.

Key facts. Cited by 173 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 75 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

58 of 75 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding20 Nov
  2. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding4 Jul
  3. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models13 May

Other references · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2019Towards VQA Models That Can Readcited by 1,696
38 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  2. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  4. 2022Token Merging: Your ViT But Fastercited by 995
  5. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  6. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  7. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  8. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  9. 2021InfographicVQAcited by 430
  10. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  11. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  12. 2021OCR-free Document Understanding Transformercited by 400
  13. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  14. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  15. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  16. 2019Scene Text Visual Question Answeringcited by 327
  17. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  18. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  19. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  20. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  21. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 170
  22. 2017Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognitioncited by 161
  23. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  24. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  25. 2019ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019cited by 138
  26. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  27. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  28. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 101
  29. 2022Towards End-to-End Unified Scene Text Detection and Layout Analysiscited by 96
  30. 2021StrucTexT: Structured Text Understanding with Multi-Modal Transformerscited by 90
  31. 2021ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spottingcited by 84
  32. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  33. 2021Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layoutscited by 82
  34. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understandingcited by 74
  35. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  36. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  37. 2023SPTS v2: Single-Point Scene Text Spottingcited by 66
  38. 2023Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solutioncited by 54

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 63; 2025: 64; 2026: 46.

173 papers.

Sort:

By area:

  1. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  3. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  4. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  5. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  6. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  7. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
  8. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  9. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  10. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  11. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  12. 2025MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsingcited by 108
38 more
  1. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  2. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
  3. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  4. 2025MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigmcited by 81
  5. 2024CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacycited by 80
  6. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 78
  7. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  8. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 70
  9. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 70
  10. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 70
  11. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
  12. 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62
  13. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  14. 2025GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensingcited by 55
  15. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  16. 2025Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detectorcited by 38
  17. 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 38
  18. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 37
  19. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  20. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  21. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  22. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  23. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 34
  24. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  25. 2025We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoningcited by 32
  26. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  27. 2024Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramidcited by 26
  28. 2024MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmarkcited by 26
  29. 2025WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?cited by 26
  30. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understandingcited by 25
  31. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Explorationcited by 25
  32. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25
  33. 2025Ocean-OCR: Towards General OCR Application via a Vision-Language Modelcited by 24
  34. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  35. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  36. 2024Token-level Correlation-guided Compression for Efficient Multimodal Document Understandingcited by 22
  37. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 22
  38. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21

50 most cited of 173 All 173 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding
  2. 2026UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents
  3. 2024From Text to Pixel: Advancing Long-Context Understanding in MLLMs
  4. 2025VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization
  5. 2025Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark
  6. 2024Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding
  7. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  8. 2025See the Text: From Tokenization to Visual Reading