mlsanityData to 14 Oct 2026

LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding

arXiv 2404.05225 ↗First posted 8 Apr 2024cs.CV cs.CLcited by 95

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Recently, leveraging large language models (LLMs) or multimodal large language models (MLLMs) for document understanding has been proven very promising. However, previous works that employ LLMs/MLLMs for document understanding have not fully explored and utilized the document layout information, which is vital for precise document understanding. In this paper, we propose LayoutLLM, an LLM/MLLM based method for document understanding. The core of LayoutLLM is a layout instruction tuning strategy, which is specially designed to enhance the comprehension and utilization of document layouts. The proposed layout instruction tuning strategy consists of two components: Layout-aware Pre-training and Layout-aware Supervised Fine-tuning. To capture the characteristics of document layout in Layout-aware Pre-training, three groups of pre-training tasks, corresponding to document-level, region-level and segment-level information, are introduced. Furthermore, a novel module called layout chain-of-thought (LayoutCoT) is devised to enable LayoutLLM to focus on regions relevant to the question and generate accurate answers. LayoutCoT is effective for boosting the performance of document understanding. Meanwhile, it brings a certain degree of interpretability, which could facilitate manual inspection and correction. Experiments on standard benchmarks show that the proposed LayoutLLM significantly outperforms existing methods that adopt open-source 7B LLMs/MLLMs for document understanding. The training data of the LayoutLLM is publicly available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/LayoutLLM

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 65 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

53 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding4 Jul
  2. 2022Bi-VLDoc: Bidirectional Vision-Language Modeling for Visually-Rich Document Understanding27 Jun

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  12. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
38 more
  1. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 520
  2. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  3. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  4. 2021OCR-free Document Understanding Transformercited by 407
  5. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  6. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  7. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  8. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  9. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  10. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 256
  11. 2017EAST: An Efficient and Accurate Scene Text Detectorcited by 248
  12. 2019Image-based table recognition: data, model, and evaluationcited by 237
  13. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 213
  14. 2021FeTaQA: Free-form Table Question Answeringcited by 175
  15. 2020DocBank: A Benchmark Dataset for Document Layout Analysiscited by 153
  16. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 151
  17. 2022DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysiscited by 148
  18. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  19. 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
  20. 2022LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understandingcited by 107
  21. 2021BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documentscited by 104
  22. 2022Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusioncited by 94
  23. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  24. 2021StrucTexT: Structured Text Understanding with Multi-Modal Transformerscited by 90
  25. 2022Unified Pretraining Framework for Document Understandingcited by 85
  26. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  27. 2022XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understandingcited by 68
  28. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  29. 2021Document AI: Benchmarks, Models and Applicationscited by 62
  30. 2023Vision Grid Transformer for Document Layout Analysiscited by 47
  31. 2023M6Doc: A Large-Scale Multi-Format, Multi-Type, Multi-Layout, Multi-Language, Multi-Annotation Category Dataset for Modern Document Layout Analysiscited by 46
  32. 2023Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluationcited by 46
  33. 2023GeoLayoutLM: Geometric Pre-training for Visual Information Extractioncited by 40
  34. 2023StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-trainingcited by 39
  35. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38
  36. 2023LMDX: Language Model-based Document Information Extraction and Localizationcited by 37
  37. 2023Attention Where It Matters: Rethinking Visual Document Understanding with Selective Region Concentrationcited by 18
  38. 2023LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understandingcited by 17

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 21; 2025: 42; 2026: 32.

95 papers.

Sort:

By area:

  1. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  2. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  3. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 67
  4. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
  5. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  6. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  7. 2025PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Frameworkcited by 40
  8. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 38
  9. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 36
  10. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  11. 2025POSTA: A Go-to Framework for Customized Artistic Poster Generationcited by 36
  12. 2025HM-RAG: Hierarchical Multi-Agent Multimodal Retrieval Augmented Generationcited by 31
38 more
  1. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  2. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  3. 2025OmniCaptioner: One Captioner to Rule Them Allcited by 20
  4. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  5. 2025When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Reviewcited by 19
  6. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18
  7. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 14
  8. 2024Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understandingcited by 12
  9. 2025MP-GUI: Modality Perception with MLLMs for GUI Understandingcited by 12
  10. 2025Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Modelscited by 12
  11. 2025IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?cited by 12
  12. 2025Invisible Prompts, Visible Threats: Malicious Font Injection in External Resources for Large Language Modelscited by 11
  13. 2025PosterSum: A Multimodal Benchmark for Scientific Poster Summarizationcited by 10
  14. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 9
  15. 2025A Simple yet Effective Layout Token in Large Language Models for Document Understandingcited by 9
  16. 2025Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segmentscited by 9
  17. 2024DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthinesscited by 8
  18. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  19. 2025PatentLMM: Large Multimodal Model for Generating Descriptions for Patent Figurescited by 7
  20. 2025LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understandingcited by 7
  21. 2026Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Trainingcited by 7
  22. 2024SAIL: Sample-Centric In-Context Learning for Document Information Extractioncited by 6
  23. 2025HIPPO: Enhancing the Table Understanding Capability of LLMs through Hybrid-Modal Preference Optimizationcited by 6
  24. 2025Problem Solved? Information Extraction Design Space for Layout-Rich Documents using LLMscited by 6
  25. 2024ProcTag: Process Tagging for Assessing the Efficacy of Document Instruction Datacited by 5
  26. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  27. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMscited by 5
  28. 2026DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splittingcited by 5
  29. 2024Hypergraph based Understanding for Document Semantic Entity Recognitioncited by 4
  30. 2024SynJAC: Synthetic-data-driven Joint-granular Adaptation and Calibration for Domain Specific Scanned Document Key Information Extractioncited by 4
  31. 2024Is Cognition Consistent with Perception? Assessing and Mitigating Multimodal Knowledge Conflicts in Document Understandingcited by 4
  32. 2025Enhancing Large Vision-Language Models with Layout Modality for Table Question Answering on Japanese Annual Securities Reportscited by 4
  33. 2025Evaluating Multimodal Large Language Models on Vertically Written Japanese Textcited by 4
  34. 2026Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligencecited by 4
  35. 2024Enhancing Mobile "How-to" Queries with Automated Search Results Verification and Rerankingcited by 3
  36. 2024See then Tell: Enhancing Key Information Extraction with Vision Groundingcited by 3
  37. 2024DocKD: Knowledge Distillation from LLMs for Open-World Document Understanding Modelscited by 3
  38. 2024GLDesigner: Leveraging Multi-Modal LLMs as Designer for Enhanced Aesthetic Text Glyph Layoutscited by 3

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026DocCogito: Aligning Layout Cognition and Step-Level Grounded Reasoning for Document Understanding
  2. 2023DocLLM: A layout-aware generative language model for multimodal document understanding
  3. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating
  4. 2025LayoutCoT: Unleashing the Deep Reasoning Potential of Large Language Models for Layout Generation
  5. 2026LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understanding
  6. 2026SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding
  7. 2025dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model
  8. 2022Knowing Where and What: Unified Word Block Pretraining for Document Understanding