mlsanityData to 14 Oct 2026

MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding

arXiv 2410.21311 ↗First posted 25 Oct 2024cs.CV cs.AIcited by 19

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have achieved remarkable performance in many vision-language tasks, yet their capabilities in fine-grained visual understanding remain insufficiently evaluated. Existing benchmarks either contain limited fine-grained evaluation samples that are mixed with other data, or are confined to object-level assessments in natural images. To holistically assess LVLMs' fine-grained visual understanding capabilities, we propose using document images with multi-granularity and multi-modal information to supplement natural images. In this light, we construct MMDocBench, a benchmark with various OCR-free document understanding tasks for the evaluation of fine-grained visual perception and reasoning abilities. MMDocBench defines 15 main tasks with 4,338 QA pairs and 11,353 supporting regions, covering various document images such as research papers, receipts, financial reports, Wikipedia tables, charts, and infographics. Based on MMDocBench, we conduct extensive experiments using 13 open-source and 3 proprietary advanced LVLMs, assessing their strengths and weaknesses across different tasks and document image types. The benchmark, task instructions, and evaluation code will be made publicly available.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 67 of its 77 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

67 of 77 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document7 Mar
  2. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding4 Jul

Other references · 65

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
38 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  11. 2016Modeling Context in Referring Expressionscited by 1,063
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  13. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  14. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  15. 2023A Survey on Multimodal Large Language Modelscited by 909
  16. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  17. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  18. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  19. 2024Yi: Open Foundation Models by 01.AIcited by 719
  20. 2020Image Segmentation Using Deep Learning: A Surveycited by 660
  21. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  22. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  23. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  24. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  25. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  26. 2021InfographicVQAcited by 437
  27. 2021OCR-free Document Understanding Transformercited by 407
  28. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  29. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  30. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  31. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  32. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  33. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  34. 2019Image-based table recognition: data, model, and evaluationcited by 237
  35. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 234
  36. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 222
  37. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  38. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 10.

19 papers.

Sort:

By area:

  1. 2025Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoningcited by 14
  2. 2025WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoningcited by 10
  3. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 9
  4. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  5. 2025NoTeS-Bank: Benchmarking Neural Transcription and Search for Scientific Notes Understandingcited by 3
  6. 2026A Computational Approach to Visual Metonymycited by 3
  7. 2025MedRepBench: A Comprehensive Benchmark for Medical Report Interpretationcited by 2
  8. 2026PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generationcited by 2
  9. 2025GenDexHand: Generative Simulation for Dexterous Handscited by 1
  10. 2026From Dead Pixels to Editable Slides: Infographic Reconstruction into Native Google Slides via Vision-Language Region Understandingcited by 1
  11. 2026DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?cited by 1
  12. 2026GUIDE: Governed Unified Intelligence for Document-to-Artifact Generation in Enterprise Settingscited by 1
7 more
  1. 2026SAGE: Governed Artifact Generation from Enterprise Guidelinescited by 1
  2. 2025Hierarchical structure understanding in complex tables with VLLMs: a benchmark and experimentscited by 0
  3. 2025PyFi: Toward Pyramid-like Financial Image Understanding for VLMs via Adversarial Agentscited by 0
  4. 2026Can VLMs Reason Robustly? A Neuro-Symbolic Investigationcited by 0
  5. 2026VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QAcited by 0
  6. 2026Office Comprehension Benchmarkcited by 0
  7. 2026PalmLeaf-VQA: A Multi-Script Visual Question Answering Benchmark for Historical Palm-Leaf Manuscript Understanding Across Diverse Regionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding
  2. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
  3. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
  4. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  5. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
  6. 2025Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark
  7. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
  8. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding