mlsanityData to 7 Oct 2026

DocFormer: End-to-End Transformer for Document Understanding

arXiv 2106.11539 ↗First posted 22 Jun 2021cs.CVcited by 254

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present DocFormer -- a multi-modal transformer based architecture for the task of Visual Document Understanding (VDU). VDU is a challenging problem which aims to understand documents in their varied formats (forms, receipts etc.) and layouts. In addition, DocFormer is pre-trained in an unsupervised fashion using carefully designed tasks which encourage multi-modal interaction. DocFormer uses text, vision and spatial features and combines them using a novel multi-modal self-attention layer. DocFormer also shares learned spatial embeddings across modalities which makes it easy for the model to correlate text to visual tokens and vice versa. DocFormer is evaluated on 4 different datasets each with strong baselines. DocFormer achieves state-of-the-art results on all of them, sometimes beating models 4x its size (in no. of parameters).

Key facts. Cited by 254 papers in the map (data through 7 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 22 Jun 2021.

Builds on

43 of 60 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
31 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  3. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  5. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  9. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  10. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  11. 2019PubLayNet: largest dataset ever for document layout analysiscited by 349
  12. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  13. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  14. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 246
  15. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  16. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  17. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  18. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  19. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  20. 2018Chargrid: Towards Understanding 2D Documentscited by 100
  21. 2020SCATTER: Selective Context Attentional Scene Text Recognizercited by 69
  22. 2020Spatial Dependency Parsing for Semi-Structured Document Information Extractioncited by 65
  23. 2019BERTgrid: Contextualized Embedding for 2D Document Representation and Understandingcited by 63
  24. 2020Kleister: A novel task for Information Extraction involving Long Documents with Complex Layoutcited by 27
  25. 2019Deep Visual Template-Free Form Parsingcited by 24
  26. 2017Cutting the Error by Half: Investigation of Very Deep CNN and Advanced Training Strategies for Document Image Classificationcited by 22
  27. 2018Document Image Classification with Intra-Domain Transfer Learning and Stacked Generalization of Deep Convolutional Neural Networkscited by 20
  28. 2020Towards Good Practices in Self-supervised Representation Learningcited by 19
  29. 2019Modular Multimodal Architecture for Document Classificationcited by 15
  30. 2017Analysis of Convolutional Neural Networks for Document Image Classificationcited by 13
  31. 2016What is the right way to represent document images?cited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 47; 2023: 65; 2024: 55; 2025: 52; 2026: 29.

254 papers.

Sort:

By area:

  1. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  3. 2021OCR-free Document Understanding Transformercited by 400
  4. 2022Multimodal Learning with Transformers: A Surveycited by 399
  5. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  6. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  7. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  8. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  9. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  10. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  11. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  12. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
38 more
  1. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  2. 2022LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understandingcited by 107
  3. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  4. 2021BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documentscited by 104
  5. 2022Towards End-to-End Unified Scene Text Detection and Layout Analysiscited by 96
  6. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  7. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 94
  8. 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
  9. 2024OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognitioncited by 74
  10. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  11. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  12. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 70
  13. 2022XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understandingcited by 68
  14. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  15. 2022FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extractioncited by 64
  16. 2021Document AI: Benchmarks, Models and Applicationscited by 61
  17. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  18. 2022MixGen: A New Multi-Modal Data Augmentationcited by 61
  19. 2023DocILE Benchmark for Document Information Localization and Extractioncited by 58
  20. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53
  21. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  22. 2023Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluationcited by 46
  23. 2023Vision Grid Transformer for Document Layout Analysiscited by 45
  24. 2021MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understandingcited by 44
  25. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
  26. 2023GeoLayoutLM: Geometric Pre-training for Visual Information Extractioncited by 40
  27. 2023StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-trainingcited by 39
  28. 2023LMDX: Language Model-based Document Information Extraction and Localizationcited by 37
  29. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  30. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  31. 2022OCR-IDL: OCR Annotations for Industry Document Library Datasetcited by 35
  32. 2022VRDU: A Benchmark for Visually-rich Document Understandingcited by 35
  33. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  34. 2024Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentationcited by 30
  35. 2022DocEnTr: An End-to-End Document Image Enhancement Transformercited by 28
  36. 2024InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructionscited by 27
  37. 2024DocVLM: Make Your VLM an Efficient Readercited by 27
  38. 2025SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinementcited by 27

50 most cited of 254 All 254 (JSON)