mlsanityData to 7 Oct 2026

OCR-free Document Understanding Transformer

arXiv 2111.15664 ↗First posted 30 Nov 2021cs.LG cs.AIcited by 400

Not on the map. Why?

Cited by 126 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Understanding document images (e.g., invoices) is a core but challenging task since it requires complex functions such as reading text and a holistic understanding of the document. Current Visual Document Understanding (VDU) methods outsource the task of reading text to off-the-shelf Optical Character Recognition (OCR) engines and focus on the understanding task with the OCR outputs. Although such OCR-based approaches have shown promising performance, they suffer from 1) high computational costs for using OCR; 2) inflexibility of OCR models on languages or types of document; 3) OCR error propagation to the subsequent process. To address these issues, in this paper, we introduce a novel OCR-free VDU model named Donut, which stands for Document understanding transformer. As the first step in OCR-free VDU research, we propose a simple architecture (i.e., Transformer) with a pre-training objective (i.e., cross-entropy loss). Donut is conceptually simple yet effective. Through extensive experiments and analyses, we show a simple OCR-free VDU model, Donut, achieves state-of-the-art performances on various VDU tasks in terms of both speed and accuracy. In addition, we offer a synthetic data generator that helps the model pre-training to be flexible in various languages and domains. The code, trained model and synthetic data are available at https://github.com/clovaai/donut.

Key facts. Cited by 400 papers in the map (data through 7 Oct 2026). Builds on 44 of its 72 references that are in the map. First posted on arXiv 30 Nov 2021.

Builds on

44 of 72 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  12. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
32 more
  1. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  2. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  3. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  4. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  5. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  6. 2019Character Region Awareness for Text Detectioncited by 273
  7. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  8. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  9. 2019Image-based table recognition: data, model, and evaluationcited by 233
  10. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  11. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  12. 2019What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysiscited by 191
  13. 2020ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Networkcited by 170
  14. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  15. 2016Robust Scene Text Recognition with Automatic Rectificationcited by 121
  16. 2016TextBoxes: A Fast Text Detector with a Single Deep Neural Networkcited by 119
  17. 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
  18. 2021BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documentscited by 104
  19. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  20. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  21. 2016Detecting Text in Natural Image with Connectionist Text Proposal Networkcited by 85
  22. 2020Spatial Dependency Parsing for Semi-Structured Document Information Extractioncited by 65
  23. 2020UnrealText: Synthesizing Realistic Scene Text Images from the Unreal Worldcited by 43
  24. 2021SynthTIGER: Synthetic Text Image GEneratoR Towards Better Text Recognition Modelscited by 43
  25. 2016Multi-Oriented Text Detection with Fully Convolutional Networkscited by 36
  26. 2020OCR Post Correction for Endangered Language Textscited by 31
  27. 2019EATEN: Entity-aware Attention for Single Shot Visual Text Extractioncited by 29
  28. 2019Deep Visual Template-Free Form Parsingcited by 24
  29. 2021ICDAR 2021 Competition on Document VisualQuestion Answeringcited by 19
  30. 2021Cost-effective End-to-end Information Extraction for Semi-structured Document Imagescited by 14
  31. 2020An Unsupervised method for OCR Post-Correction and Spelling Normalisation for Finnishcited by 9
  32. 2021DocReader: Bounding-Box Free Training of a Document Information Extraction Modelcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 50; 2024: 109; 2025: 106; 2026: 126.

400 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  5. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  6. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  7. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  8. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  9. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  10. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  11. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  12. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
38 more
  1. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  2. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  3. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  4. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  5. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  6. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  7. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  8. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  9. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  10. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  11. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  12. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  13. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  14. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 132
  15. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  16. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 120
  17. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  18. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  19. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
  20. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  21. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  22. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  23. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 94
  24. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  25. 2025LongCat-Image Technical Reportcited by 92
  26. 2024Unveiling Encoder-Free Vision-Language Modelscited by 90
  27. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  28. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  29. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  30. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
  31. 2025MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigmcited by 81
  32. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  33. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  34. 2024OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognitioncited by 74
  35. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 72
  36. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 70
  37. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  38. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62

50 most cited of 400 All 400 (JSON)