mlsanityData to 7 Oct 2026

A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding

arXiv 2407.01976 ↗First posted 2 Jul 2024cs.CL cs.AI cs.MMcited by 57

Not on the map. Why?

Abstract (arXiv)

Recently, many studies have demonstrated that exclusively incorporating OCR-derived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce Interleaving Layout and Text in a Large Language Model (LayTextLLM)} for document understanding. LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in KIE and VQA. Comprehensive benchmark evaluations reveal significant improvements of LayTextLLM, with a 15.2% increase on KIE tasks and 10.7% on VQA tasks compared to previous SOTA OCR-based LLMs. All resources are available at https://github.com/LayTextLLM/LayTextLLM.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 60 of its 74 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

60 of 74 references.

References · 60

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  12. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
38 more
  1. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  5. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  6. 2024Yi: Open Foundation Models by 01.AIcited by 706
  7. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  10. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  11. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  12. 2021InfographicVQAcited by 430
  13. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  14. 2021OCR-free Document Understanding Transformercited by 400
  15. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  16. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  17. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  18. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  19. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  20. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  21. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  22. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  23. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  24. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  25. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  26. 2020DocBank: A Benchmark Dataset for Document Layout Analysiscited by 152
  27. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  28. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  29. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  30. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  31. 2021BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documentscited by 104
  32. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 94
  33. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  34. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understandingcited by 74
  35. 2020Spatial Dependency Parsing for Semi-Structured Document Information Extractioncited by 65
  36. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
  37. 2023DocILE Benchmark for Document Information Localization and Extractioncited by 58
  38. 2023Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solutioncited by 54

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 23; 2026: 31.

57 papers.

Sort:

By area:

  1. 2025Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoningcited by 34
  2. 2025WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?cited by 26
  3. 2025Advancing Sequential Numerical Prediction in Autoregressive Modelscited by 26
  4. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  5. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  6. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  7. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  8. 2025RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuningcited by 10
  9. 2025A Simple yet Effective Layout Token in Large Language Models for Document Understandingcited by 9
  10. 2024DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthinesscited by 8
  11. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  12. 2024SAIL: Sample-Centric In-Context Learning for Document Information Extractioncited by 6
38 more
  1. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  2. 2025Task-Oriented 6-DoF Grasp Pose Detection in Clutterscited by 4
  3. 2025Enhancing Large Vision-Language Models with Layout Modality for Table Question Answering on Japanese Annual Securities Reportscited by 4
  4. 2026Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligencecited by 4
  5. 2026Revise: A Framework for Revising OCRed text in Practical Information Systems with Data Contamination Strategycited by 3
  6. 2025PILOT: A Promptable Interleaved Layout-aware OCR Transformercited by 2
  7. 2025Towards Reliable and Interpretable Document Question Answering via VLMscited by 2
  8. 2025ARIAL: An Agentic Framework for Document VQA with Precise Answer Localizationcited by 2
  9. 2025TDRI: Two-Phase Dialogue Refinement and Co-Adaptation for Interactive Image Generationcited by 1
  10. 2025MGA-VQA: Secure and Interpretable Graph-Augmented Visual Question Answering with Memory-Guided Protection Against Unauthorized Knowledge Usecited by 1
  11. 2025DocVAL: Validated Chain-of-Thought Distillation for Grounded Document VQAcited by 1
  12. 20254D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillationcited by 1
  13. 2026LingDT-VL-OCR: Structure-Aware Document-Level Parsing with Fine-Grained Visual Referencecited by 1
  14. 2026Q-Mask: Query-driven Causal Masks for Text Anchoring in OCR-Oriented Vision-Language Modelscited by 1
  15. 2025First-place Solution for Streetscape Shop Sign Recognition Competitioncited by 0
  16. 2025Cross-Modal Synergies: Unveiling the Potential of Motion-Aware Fusion Networks in Handling Dynamic and Static ReID Scenarioscited by 0
  17. 2025CalliReader: Contextualizing Chinese Calligraphy via an Embedding-Aligned Vision-Language Modelcited by 0
  18. 2025OMR-Diffusion:Optimizing Multi-Round Enhanced Training in Diffusion Models for Improved Intent Understandingcited by 0
  19. 2026LPCAN: Lightweight Pyramid Cross-Attention Network for Rail Surface Defect Detection Using RGB-D Datacited by 0
  20. 2026Knowledge-Embedded and Hypernetwork-Guided Few-Shot Substation Meter Defect Image Generation Methodcited by 0
  21. 2026PubMed-OCR: PMC Open Access OCR Annotationscited by 0
  22. 2026Up to 36x Speedup: Mask-based Parallel Inference Paradigm for Key Information Extraction in MLLMscited by 0
  23. 2026MA-LipNet: Multi-Dimensional Attention Networks for Robust Lipreadingcited by 0
  24. 2026The Diffusion Duet: Harmonizing Dual Channels with Wavelet Suppression for Image Separationcited by 0
  25. 2026Detecting Deepfakes with Multivariate Soft Blending and CLIP-based Image-Text Alignmentcited by 0
  26. 2026Efficient Image Super-Resolution with Multi-Scale Spatial Adaptive Attention Networkscited by 0
  27. 2026A Deformable Attention-Based Detection Transformer with Cross-Scale Feature Fusion for Industrial Coil Spring Inspectioncited by 0
  28. 2026Caption Generation for Dongba Paintings via Prompt Learning and Semantic Fusioncited by 0
  29. 2026Transmittance-Guided Structure-Texture Decomposition for Nighttime Image Dehazingcited by 0
  30. 2026Hierarchical Awareness Adapters with Hybrid Pyramid Feature Fusion for Dense Depth Predictioncited by 0
  31. 2026Adaptive Slicing-Assisted Hyper Inference for Enhanced Small Object Detection in High-Resolution Imagerycited by 0
  32. 2026Feature Perturbation Pool-based Fusion Network for Unified Multi-Class Industrial Defect Detectioncited by 0
  33. 2026Lightweight Real-Time Rendering Parameter Optimization via XGBoost-Driven Lookup Tablescited by 0
  34. 2026Image Classification via Random Dilated Convolution with Multi-Branch Feature Extraction and Context Excitationcited by 0
  35. 2026Gait Recognition via Deep Residual Networks and Multi-Branch Feature Fusioncited by 0
  36. 2026Multi-Branch Non-Homogeneous Image Dehazing via Concentration Partitioning and Image Fusioncited by 0
  37. 2026Local Spatiotemporal Convolutional Network for Robust Gait Recognitioncited by 0
  38. 2026ForMaT: Dataset for Visually-Grounded Multilingual PDF Translationcited by 0

50 most cited of 57 All 57 (JSON)