mlsanityData to 14 Oct 2026

DOGR: Towards Versatile Visual Document Grounding and Referring

arXiv 2411.17125 ↗First posted 26 Nov 2024cs.CV cs.AIcited by 10

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

With recent advances in Multimodal Large Language Models (MLLMs), grounding and referring capabilities have gained increasing attention for achieving detailed understanding and flexible user interaction. However, these capabilities still remain underdeveloped in visual document understanding due to the scarcity of fine-grained datasets and comprehensive benchmarks. To fill this gap, we propose the DOcument Grounding and Referring data engine (DOGR-Engine), which generates two types of high-quality fine-grained document data: (1) multi-granular parsing data to improve text localization and recognition, and (2) instruction-tuning data to activate MLLMs' grounding and referring capabilities in dialogue and reasoning. Using the DOGR-Engine, we construct DOGR-Bench, a benchmark covering seven grounding and referring tasks across three document types (chart, poster, and PDF document), offering a comprehensive evaluation of fine-grained document understanding. Leveraging the generated data, we further develop DOGR, a strong baseline model that excels in text localization and recognition, while precisely grounds and refers to key textual information during conversation and reasoning, thereby advancing document understanding to a finer granularity and enable flexible interaction paradigms.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 62 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

45 of 62 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models11 Dec
  2. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models5 Dec
  3. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun

Other references · 40

  1. 2024GPT-4o System Cardcited by 4,588
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  5. 2024Qwen2 Technical Reportcited by 2,120
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  10. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  12. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
28 more
  1. 2016A Diagram Is Worth A Dozen Imagescited by 804
  2. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  3. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  4. 2021OCR-free Document Understanding Transformercited by 407
  5. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  6. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  7. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  8. 2019Scene Text Visual Question Answeringcited by 331
  9. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 267
  10. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  11. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  12. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  13. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 151
  14. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 150
  15. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  16. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 120
  17. 2021HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generationcited by 118
  18. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 114
  19. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 112
  20. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  21. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 96
  22. 2021CanvasVAE: Learning to Generate Vector Graphic Documentscited by 95
  23. 2021Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layoutscited by 82
  24. 2020Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Modelcited by 78
  25. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  26. 2022Syntax-Aware Network for Handwritten Mathematical Expression Recognitioncited by 68
  27. 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62
  28. 2023RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbationscited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 23
  2. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  3. 2025DocR1: Evidence Page-Guided GRPO for Multi-Page Document Understandingcited by 11
  4. 2026Prune as You Generate: Online Rollout Pruning for Faster and Better RLVRcited by 10
  5. 2026DocPrune:Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruningcited by 4
  6. 2025VL4Gaze: Unleashing Vision-Language Models for Gaze Followingcited by 3
  7. 2026ChartREG++: Towards Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referringcited by 2
  8. 2026MathDoc: Benchmarking Structured Extraction and Active Refusal on Noisy Mathematics Exam Paperscited by 0
  9. 2026An LMM for Precisely Grounding Elements in Documentscited by 0
  10. 2026Stop Thinking, Start Looking: Efficient Post-Training for Multimodal Document Question Answering via Reasoning-Free Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMs
  2. 2025Towards Visual Text Grounding of Multimodal Large Language Model
  3. 2025DRISHTIKON: Visual Grounding at Multiple Granularities in Documents
  4. 2026DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding
  5. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
  6. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating
  7. 2026Chain-of-Caption: Training-free improvement of multimodal large language model on referring expression comprehension
  8. 2026DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents