mlsanityData to 7 Oct 2026

Unifying Vision, Text, and Layout for Universal Document Processing

arXiv 2212.02623 ↗First posted 5 Dec 2022cs.CV cs.AI cs.CLcited by 140

Not on the map. Why?

Cited by 27 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose Universal Document Processing (UDOP), a foundation Document AI model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation. UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation. With a novel Vision-Text-Layout Transformer, UDOP unifies pretraining and multi-domain downstream tasks into a prompt-based sequence generation scheme. UDOP is pretrained on both large-scale unlabeled document corpora using innovative self-supervised objectives and diverse labeled data. UDOP also learns to generate document images from text and layout modalities via masked image reconstruction. To the best of our knowledge, this is the first time in the field of document AI that one model simultaneously achieves high-quality neural document editing and content customization. Our method sets the state-of-the-art on 8 Document AI tasks, e.g., document understanding and QA, across diverse data domains like finance reports, academic papers, and websites. UDOP ranks first on the leaderboard of the Document Understanding Benchmark.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Builds on 49 of its 59 references that are in the map. First posted on arXiv 5 Dec 2022.

Builds on

49 of 59 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
37 more
  1. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  2. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  3. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  4. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  5. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  6. 2021InfographicVQAcited by 430
  7. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  8. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  9. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  10. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  11. 2021OCR-free Document Understanding Transformercited by 400
  12. 2019PubLayNet: largest dataset ever for document layout analysiscited by 349
  13. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  14. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  15. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  16. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  17. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  18. 2020DocBank: A Benchmark Dataset for Document Layout Analysiscited by 152
  19. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  20. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  21. 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
  22. 2022LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understandingcited by 107
  23. 2021BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documentscited by 104
  24. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  25. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  26. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  27. 2021StrucTexT: Structured Text Understanding with Multi-Modal Transformerscited by 90
  28. 2022Unified Pretraining Framework for Document Understandingcited by 85
  29. 2021Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layoutscited by 82
  30. 2022XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understandingcited by 68
  31. 2022FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extractioncited by 64
  32. 2020LAMBERT: Layout-Aware (Language) Modeling for information extractioncited by 53
  33. 2020AxCell: Automatic Extraction of Results from Machine Learning Paperscited by 48
  34. 2022i-Code: An Integrative and Composable Multimodal Learning Frameworkcited by 24
  35. 2020Towards a Multi-modal, Multi-task Learning based Pre-training Framework for Document Representation Learningcited by 21
  36. 2021LAMPRET: Layout-Aware Multimodal PreTraining for Document Understandingcited by 15
  37. 2022Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attentioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 44; 2025: 38; 2026: 34.

140 papers.

Sort:

By area:

  1. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  2. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  3. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  4. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  5. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  6. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  7. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  8. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  9. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  10. 2025MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsingcited by 108
  11. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  12. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
38 more
  1. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  2. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 78
  3. 2024OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognitioncited by 74
  4. 2023DocILE Benchmark for Document Information Localization and Extractioncited by 58
  5. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  6. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  7. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 51
  8. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  9. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  10. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  11. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 34
  12. 2024DocRes: A Generalist Model Toward Unifying Document Image Restoration Taskscited by 27
  13. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understandingcited by 25
  14. 2024LayoutLLM: Large Language Model Instruction Tuning for Visually Rich Document Understandingcited by 24
  15. 2024GRAM: Global Reasoning for Multi-Page VQAcited by 23
  16. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  17. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 22
  18. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  19. 2026Dolphin-v2: Universal Document Parsing via Scalable Anchor Promptingcited by 20
  20. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  21. 2023Attention Where It Matters: Rethinking Visual Document Understanding with Selective Region Concentrationcited by 18
  22. 2025AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMscited by 18
  23. 2023UPOCR: Towards Unified Pixel-Level OCR Interfacecited by 12
  24. 2024RoDLA: Benchmarking the Robustness of Document Layout Analysis Modelscited by 12
  25. 2024Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understandingcited by 12
  26. 2024Hierarchical Visual Feature Aggregation for OCR-Free Document Understandingcited by 12
  27. 2023Beyond Document Page Classification: Design, Datasets, and Challengescited by 10
  28. 2025Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrievalcited by 10
  29. 2024LAPDoc: Layout-Aware Prompting for Documentscited by 9
  30. 2024JDocQA: Japanese Document Question Answering Dataset for Generative Language Modelscited by 9
  31. 2024BuDDIE: A Business Document Dataset for Multi-task Information Extractioncited by 8
  32. 2024VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document Understandingcited by 8
  33. 2024DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthinesscited by 8
  34. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  35. 2023Analyzing the Efficacy of an LLM-Only Approach for Image-based Document Question Answeringcited by 7
  36. 2024μgat: Improving Single-Page Document Parsing by Providing Multi-Page Contextcited by 7
  37. 2024Deep Learning based Key Information Extraction from Business Documents: Systematic Literature Reviewcited by 6
  38. 2025Visual Template Inference for Data Extraction from Documentscited by 6

50 most cited of 140 All 140 (JSON)