mlsanityData to 7 Oct 2026

DocLLM: A layout-aware generative language model for multimodal document understanding

arXiv 2401.00908 ↗First posted 31 Dec 2023cs.CLcited by 112

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Enterprise documents such as forms, invoices, receipts, reports, contracts, and other similar records, often carry rich semantics at the intersection of textual and spatial modalities. The visual cues offered by their complex layouts play a crucial role in comprehending these documents effectively. In this paper, we present DocLLM, a lightweight extension to traditional large language models (LLMs) for reasoning over visual documents, taking into account both textual semantics and spatial layout. Our model differs from existing multimodal LLMs by avoiding expensive image encoders and focuses exclusively on bounding box information to incorporate the spatial layout structure. Specifically, the cross-alignment between text and spatial modalities is captured by decomposing the attention mechanism in classical transformers to a set of disentangled matrices. Furthermore, we devise a pre-training objective that learns to infill text segments. This approach allows us to address irregular layouts and heterogeneous content frequently encountered in visual documents. The pre-trained model is fine-tuned using a large-scale instruction dataset, covering four core document intelligence tasks. We demonstrate that our solution outperforms SotA LLMs on 14 out of 16 datasets across all tasks, and generalizes well to 4 out of 5 previously unseen datasets.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Builds on 62 of its 79 references that are in the map. First posted on arXiv 31 Dec 2023.

Builds on

62 of 79 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2020GLU Variants Improve Transformercited by 1,724
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  10. 2019Root Mean Square Layer Normalizationcited by 1,425
  11. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  12. 2023PaLM 2 Technical Reportcited by 1,154
  13. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  14. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  15. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  16. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  17. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  18. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  19. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  20. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  21. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  22. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  23. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  24. 2021OCR-free Document Understanding Transformercited by 400
  25. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  26. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  27. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  28. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  29. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  30. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  31. 2023Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoningcited by 177
  32. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  33. 2020DocBank: A Benchmark Dataset for Document Layout Analysiscited by 152
  34. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  35. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  36. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  37. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  38. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 48; 2026: 36.

112 papers.

Sort:

By area:

  1. 2024Large Language Models: A Surveycited by 506
  2. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  3. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  4. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
  5. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  6. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 86
  7. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 78
  8. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  9. 2024Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extractioncited by 44
  10. 2025dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Modelcited by 44
  11. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  12. 2024FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Modelscited by 39
38 more
  1. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  2. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 34
  3. 2024DocVLM: Make Your VLM an Efficient Readercited by 27
  4. 2024Towards Multimodal In-Context Learning for Vision & Language Modelscited by 24
  5. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  6. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  7. 2025Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligencecited by 20
  8. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  9. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  10. 2025Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understandingcited by 18
  11. 2024VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Textcited by 15
  12. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  13. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  14. 2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understandingcited by 13
  15. 2024Visually Guided Generative Text-Layout Pre-training for Document Intelligencecited by 12
  16. 2024Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understandingcited by 12
  17. 2025The Confidence Paradox: Can LLM Know When It's Wrongcited by 11
  18. 2024ANLS* -- A Universal Document Processing Metric for Generative Large Language Modelscited by 10
  19. 2025PosterSum: A Multimodal Benchmark for Scientific Poster Summarizationcited by 10
  20. 2024LAPDoc: Layout-Aware Prompting for Documentscited by 9
  21. 2025A Simple yet Effective Layout Token in Large Language Models for Document Understandingcited by 9
  22. 2025Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segmentscited by 9
  23. 2024BuDDIE: A Business Document Dataset for Multi-task Information Extractioncited by 8
  24. 2024VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document Understandingcited by 8
  25. 2025BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotationscited by 8
  26. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  27. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  28. 2024Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Reportcited by 7
  29. 2025AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understandingcited by 7
  30. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMscited by 7
  31. 2025Table as a Modality for Large Language Modelscited by 7
  32. 2024Opportunities and Challenges of Generative-AI in Financecited by 6
  33. 2024SAIL: Sample-Centric In-Context Learning for Document Information Extractioncited by 6
  34. 2025Problem Solved? Information Extraction Design Space for Layout-Rich Documents using LLMscited by 6
  35. 2026Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCRcited by 6
  36. 2026MultiDocFusion: Hierarchical and Multimodal Chunking Pipeline for Enhanced RAG on Long Industrial Documentscited by 6
  37. 2026DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understandingcited by 6
  38. 2025Hybrid OCR-LLM Framework for Enterprise-Scale Document Information Extraction Under Copy-heavy Taskcited by 5

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoning
  2. 2026DocCogito: Aligning Layout Cognition and Step-Level Grounded Reasoning for Document Understanding
  3. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding
  4. 2026SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding
  5. 2024Read and Think: An Efficient Step-wise Multimodal Language Model for Document Understanding and Reasoning
  6. 2026Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding
  7. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval
  8. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating