mlsanityData to 7 Oct 2026

Deep Learning based Visually Rich Document Content Understanding: A Survey

arXiv 2408.01287 ↗First posted 2 Aug 2024cs.CL cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

Visually Rich Documents (VRDs) play a vital role in domains such as academia, finance, healthcare, and marketing, as they convey information through a combination of text, layout, and visual elements. Traditional approaches to extracting information from VRDs rely heavily on expert knowledge and manual annotation, making them labor-intensive and inefficient. Recent advances in deep learning have transformed this landscape by enabling multimodal models that integrate vision, language, and layout features through pretraining, significantly improving information extraction performance. This survey presents a comprehensive overview of deep learning-based frameworks for VRD Content Understanding (VRD-CU). We categorize existing methods based on their modeling strategies and downstream tasks, and provide a comparative analysis of key components, including feature representation, fusion techniques, model architectures, and pretraining objectives. Additionally, we highlight the strengths and limitations of each approach and discuss their suitability for different applications. The paper concludes with a discussion of current challenges and emerging trends, offering guidance for future research and practical deployment in real-world scenarios.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 143 of its 176 references that are in the map. First posted on arXiv 2 Aug 2024.

Builds on

143 of 176 references.

References · 143

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  11. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2020End-to-End Object Detection with Transformerscited by 7,384
  2. 2017Mask R-CNNcited by 7,259
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  11. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  13. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  14. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  15. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  16. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  17. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  18. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  19. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  20. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  21. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  22. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  23. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  24. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  25. 2017Selective Classification for Deep Neural Networkscited by 799
  26. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  27. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  28. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  29. 2016Neural Architectures for Named Entity Recognitioncited by 548
  30. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  31. 2018TADAM: Task dependent adaptive metric for improved few-shot learningcited by 408
  32. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  33. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  34. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  35. 2021OCR-free Document Understanding Transformercited by 400
  36. 2019CSPNet: A New Backbone that can Enhance Learning Capability of CNNcited by 381
  37. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  38. 2019PubLayNet: largest dataset ever for document layout analysiscited by 349

50 most cited of 143 All 143 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 9.

21 papers.

Sort:

By area:

  1. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 72
  2. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  3. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  4. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  5. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  6. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  7. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  8. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  9. 2024SynJAC: Synthetic-data-driven Joint-granular Adaptation and Calibration for Domain Specific Scanned Document Key Information Extractioncited by 4
  10. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  11. 2026UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documentscited by 4
  12. 2026Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQAcited by 4
9 more
  1. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Surveycited by 2
  2. 2026MINER: Mining Multimodal Internal Representation for Efficient Retrievalcited by 2
  3. 2026TextRich: A Multi-Domain Benchmark for Detecting AI-Generated Text-Rich Images from GPT-Image-2cited by 2
  4. 2025Multi-Modal Vision vs. Text-Based Parsing: Benchmarking LLM Strategies for Invoice Processingcited by 1
  5. 2026Enhancing Large Multimodal Models in Key Information Extraction via Scene-Aware Document Synthesiscited by 1
  6. 2025Hierarchical Document Parsing via Large Margin Feature Matching and Heuristicscited by 0
  7. 2026Multimodal Approaches for Visually-Rich Document Type Classification: A Comparative Analysiscited by 0
  8. 2026LAVA: Logic-Aware Validation and Augmentation Framework for Large-Scale Financial Document Auditingcited by 0
  9. 2026The Right Information Extraction Pipeline Depends on the Document: Accuracy-Energy Trade-offs for Small, Local Modelscited by 0