mlsanityData to 7 Oct 2026

LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking

arXiv 2204.08387 ↗First posted 18 Apr 2022cs.CL cs.CVcited by 481

Not on the map. Why?

Cited by 73 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Self-supervised pre-training techniques have achieved remarkable progress in Document AI. Most multimodal pre-trained models use a masked language modeling objective to learn bidirectional representations on the text modality, but they differ in pre-training objectives for the image modality. This discrepancy adds difficulty to multimodal representation learning. In this paper, we propose LayoutLMv3 to pre-train multimodal Transformers for Document AI with unified text and image masking. Additionally, LayoutLMv3 is pre-trained with a word-patch alignment objective to learn cross-modal alignment by predicting whether the corresponding image patch of a text word is masked. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks. Experimental results show that LayoutLMv3 achieves state-of-the-art performance not only in text-centric tasks, including form understanding, receipt understanding, and document visual question answering, but also in image-centric tasks such as document image classification and document layout analysis. The code and models are publicly available at urlhttps://aka.ms/layoutlmv3.

Key facts. Cited by 481 papers in the map (data through 7 Oct 2026). Builds on 54 of its 59 references that are in the map. First posted on arXiv 18 Apr 2022.

Builds on

54 of 59 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  5. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  6. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  9. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  10. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  11. 2016Neural Architectures for Named Entity Recognitioncited by 548
  12. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  13. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  14. 2019PubLayNet: largest dataset ever for document layout analysiscited by 349
  15. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  16. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  17. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  18. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  19. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  20. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  21. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  22. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  23. 2021Benchmarking Detection Transfer Learning with Vision Transformerscited by 130
  24. 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
  25. 2022LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understandingcited by 107
  26. 2021BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documentscited by 104
  27. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  28. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  29. 2021StrucTexT: Structured Text Understanding with Multi-Modal Transformerscited by 90
  30. 2019Graph Convolution for Multimodal Information Extraction from Visually Rich Documentscited by 88
  31. 2022XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understandingcited by 68
  32. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-trainingcited by 67
  33. 2021Towards Robust Visual Information Extraction in Real World: New Dataset and Novel Solutioncited by 64
  34. 2022FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extractioncited by 64
  35. 2020TRIE: End-to-End Text Reading and Information Extraction for Document Understandingcited by 61
  36. 2021Document AI: Benchmarks, Models and Applicationscited by 61
  37. 2020X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformerscited by 58
  38. 2020LAMBERT: Layout-Aware (Language) Modeling for information extractioncited by 53

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 25; 2023: 94; 2024: 116; 2025: 119; 2026: 127.

481 papers.

Sort:

By area:

  1. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  2. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  3. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  4. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  5. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  6. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  7. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  8. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  9. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  10. 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 144
  11. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  12. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
38 more
  1. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 132
  2. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  3. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  4. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
  5. 2025MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsingcited by 108
  6. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  7. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  8. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  9. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  10. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 94
  11. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 82
  12. 2025MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigmcited by 81
  13. 2024DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perceptioncited by 80
  14. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 78
  15. 2024OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognitioncited by 74
  16. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 70
  17. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  18. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  19. 2023DocILE Benchmark for Document Information Localization and Extractioncited by 58
  20. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  21. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  22. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53
  23. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 51
  24. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  25. 2023Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluationcited by 46
  26. 2023Vision Grid Transformer for Document Layout Analysiscited by 45
  27. 2024Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extractioncited by 44
  28. 2025dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Modelcited by 44
  29. 2023GeoLayoutLM: Geometric Pre-training for Visual Information Extractioncited by 40
  30. 2023StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-trainingcited by 39
  31. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38
  32. 2023LMDX: Language Model-based Document Information Extraction and Localizationcited by 37
  33. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  34. 2022VRDU: A Benchmark for Visually-rich Document Understandingcited by 35
  35. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  36. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 34
  37. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 33
  38. 2024MobileFlow: A Multimodal LLM For Mobile GUI Agentcited by 31

50 most cited of 481 All 481 (JSON)