mlsanityData to 14 Oct 2026

DocVLM: Make Your VLM an Efficient Reader

arXiv 2412.08746 ↗First posted 11 Dec 2024cs.CV cs.LGcited by 28

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Vision-Language Models (VLMs) excel in diverse visual tasks but face challenges in document understanding, which requires fine-grained text processing. While typical visual tasks perform well with low-resolution inputs, reading-intensive applications demand high-resolution, resulting in significant computational overhead. Using OCR-extracted text in VLM prompts partially addresses this issue but underperforms compared to full-resolution counterpart, as it lacks the complete visual context needed for optimal performance. We introduce DocVLM, a method that integrates an OCR-based modality into VLMs to enhance document processing while preserving original weights. Our approach employs an OCR encoder to capture textual content and layout, compressing these into a compact set of learned queries incorporated into the VLM. Comprehensive evaluations across leading VLMs show that DocVLM significantly reduces reliance on high-resolution images for document understanding. In limited-token regimes (448×448), DocVLM with 64 learned queries improves DocVQA results from 56.0% to 86.6% when integrated with InternVL2 and from 84.4% to 91.2% with Qwen2-VL. In LLaVA-OneVision, DocVLM achieves improved results while using 80% less image tokens. The reduced token usage allows processing multiple pages effectively, showing impressive zero-shot results on DUDE and state-of-the-art performance on MP-DocVQA, highlighting DocVLM's potential for applications requiring high-performance and efficiency.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 62 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

57 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models13 May

Other references · 56

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  11. 2019Towards VQA Models That Can Readcited by 1,718
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
38 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  2. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  5. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  6. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  7. 2021InfographicVQAcited by 437
  8. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  9. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  10. 2019Scene Text Visual Question Answeringcited by 331
  11. 2024What matters when building vision-language models?cited by 326
  12. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 256
  13. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  14. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  15. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  16. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  17. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  18. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 135
  19. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 126
  20. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 116
  21. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 114
  22. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 105
  23. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 98
  24. 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
  25. 2022DeepSolo: Let Transformer Decoder with Explicit Points Solo for Text Spottingcited by 74
  26. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  27. 2020Sequence-to-Sequence Contrastive Learning for Text Recognitioncited by 70
  28. 2020SCATTER: Selective Context Attentional Scene Text Recognizercited by 69
  29. 2022DPText-DETR: Towards Better Scene Text Detection with Dynamic Points in Transformercited by 50
  30. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  31. 2023DTrOCR: Decoder-only Transformer for Optical Character Recognitioncited by 40
  32. 2022OCR-IDL: OCR Annotations for Industry Document Library Datasetcited by 35
  33. 2022GLASS: Global to Local Attention for Scene-Text Spottingcited by 34
  34. 2024Question Aware Vision Transformer for Multimodal Reasoningcited by 32
  35. 2023Towards Unified Scene Text Spotting based on Sequence Generationcited by 27
  36. 2024InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructionscited by 27
  37. 2023Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answeringcited by 24
  38. 2024GRAM: Global Reasoning for Multi-Page VQAcited by 23

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 18.

28 papers.

Sort:

By area:

  1. 2025FineVision: Open Data Is All You Needcited by 53
  2. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 9
  3. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  4. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 7
  5. 2025SCAN: Semantic Document Layout Analysis for Textual and Visual Retrieval-Augmented Generationcited by 5
  6. 2026UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documentscited by 5
  7. 2025Hidden Tail: Adversarial Image Causing Stealthy Resource Consumption in Vision-Language Modelscited by 3
  8. 2025Towards Reliable and Interpretable Document Question Answering via VLMscited by 2
  9. 2025DocSLM: A Small Vision-Language Model for Long Multimodal Document Understandingcited by 2
  10. 2026DODO: Discrete OCR Diffusion Modelscited by 2
  11. 2026Hierarchical Evidence-Driven Reasoning for Long Document Understandingcited by 2
  12. 2025Exploring OCR-augmented Generation for Bilingual VQAcited by 1
16 more
  1. 2026From Dead Pixels to Editable Slides: Infographic Reconstruction into Native Google Slides via Vision-Language Region Understandingcited by 1
  2. 2026Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Familiescited by 1
  3. 2026Operationalizing Document AI: A Microservice Architecture for OCR and LLM Pipelines in Productioncited by 1
  4. 2025Bridging Vision, Language, and Mathematics: Pictographic Character Reconstruction with B'ezier Curvescited by 0
  5. 2025Optimizing Multimodal Language Models through Attention-based Interpretabilitycited by 0
  6. 2025Spatially-Grounded Document Retrieval via Patch-to-Region Relevance Propagationcited by 0
  7. 2026Evaluating VLMs' Spatial Reasoning Over Robot Motion: A Step Towards Robot Planning with Motion Preferencescited by 0
  8. 2026Evo-Retriever: LLM-Guided Curriculum Evolution with Viewpoint-Pathway Collaboration for Multimodal Document Retrievalcited by 0
  9. 2026DISCO: Document Intelligence Suite for COmparative Evaluationcited by 0
  10. 2026A Dataset for Dynamic Human Preferences for Vision Language Modelscited by 0
  11. 2026Edges Before Embeddings: A Confidence-Aware Blur Gate for Vision-Language Pipelinescited by 0
  12. 2026MatPhaseBench: A Semantics-Guided Benchmark for Materials Phase Diagrams Understandingcited by 0
  13. 2026State-Conditioned Visual Evidence Retrieval for Fine-Grained Perception in Document Vision-Language Modelscited by 0
  14. 2026Lot Machine: Multimodal Lot Extraction from Auction Catalogscited by 0
  15. 2026Transcribe, Then Reason: Two-Pass Decomposition for Multimodal Reviewcited by 0
  16. 2026Found but Not Read: When Extracted Text Closes the Retrieval-Reading Gap in Document Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Building and better understanding vision-language models: insights and future directions
  2. 2024TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models
  3. 2024HRVDA: High-Resolution Visual Document Assistant
  4. 2026Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models
  5. 2026LensVLM: Selective Context Expansion for Compressed Visual Representation of Text
  6. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends
  7. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  8. 2025Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality