mlsanityData to 14 Oct 2026

Towards Visual Text Grounding of Multimodal Large Language Model

arXiv 2504.04974 ↗First posted 7 Apr 2025cs.CV cs.AI cs.CLcited by 18

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Despite the existing evolution of Multimodal Large Language Models (MLLMs), a non-neglectable limitation remains in their struggle with visual text grounding, especially in text-rich images of documents. Document images, such as scanned forms and infographics, highlight critical challenges due to their complex layouts and textual content. However, current benchmarks do not fully address these challenges, as they mostly focus on visual grounding on natural images, rather than text-rich document images. Thus, to bridge this gap, we introduce TRIG, a novel task with a newly designed instruction dataset for benchmarking and improving the Text-Rich Image Grounding capabilities of MLLMs in document question-answering. Specifically, we propose an OCR-LLM-human interaction pipeline to create 800 manually annotated question-answer pairs as a benchmark and a large-scale training set of 90 synthetic data based on four diverse datasets. A comprehensive evaluation of various MLLMs on our proposed benchmark exposes substantial limitations in their grounding capability on text-rich images. In addition, we propose two simple and effective TRIG methods based on general instruction tuning and plug-and-play efficient embedding, respectively. By finetuning MLLMs on our synthetic dataset, they promisingly improve spatial reasoning and grounding capabilities.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 63 of its 80 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

63 of 80 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension4 Dec
  2. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement24 May
  3. 2024Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models28 Mar
  4. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions20 Feb
  5. 2023Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMs22 Nov
  6. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  8. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  9. 2023World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models14 Jun
  10. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr
  11. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov

Other references · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  11. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  12. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
38 more
  1. 2021Grounded Language-Image Pre-trainingcited by 1,156
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  5. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  7. 2022Large Language Models Can Self-Improvecited by 664
  8. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  9. 2021InfographicVQAcited by 437
  10. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  11. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  12. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  13. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  14. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  15. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  16. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  17. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  18. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  19. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 251
  20. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  21. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  22. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  23. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  24. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  25. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 96
  26. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 94
  27. 2022SVTR: Scene Text Recognition with a Single Visual Modelcited by 88
  28. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  29. 2021LayoutReader: Pre-training of Text and Layout for Reading Order Detectioncited by 80
  30. 2016A Joint Speaker-Listener-Reinforcer Model for Referring Expressionscited by 74
  31. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understandingcited by 74
  32. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  33. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  34. 2017Comprehension-guided referring expressionscited by 52
  35. 2024"We Need Structured Output": Towards User-centered Constraints on Large Language Model Outputcited by 46
  36. 2023StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-trainingcited by 39
  37. 2020A Survey of Deep Learning Approaches for OCR and Document Understandingcited by 32
  38. 2024RuleR: Improving LLM Controllability by Rule-based Data Recyclingcited by 11

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 5.

18 papers.

Sort:

By area:

  1. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 145
  2. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  3. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 14
  4. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 10
  5. 2025MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustnesscited by 5
  6. 2025CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoningcited by 4
  7. 2026DocPrune:Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruningcited by 4
  8. 2025From Image Generation to Infrastructure Design: a Multi-agent Pipeline for Street Design Generationcited by 3
  9. 2026ReAlign: Optimizing the Visual Document Retriever with Reasoning-Guided Fine-Grained Alignmentcited by 3
  10. 2025DRISHTIKON: Visual Grounding at Multiple Granularities in Documentscited by 1
  11. 2025MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlookcited by 1
  12. 2025DocVAL: Validated Chain-of-Thought Distillation for Grounded Document VQAcited by 1
6 more
  1. 2026Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Modelscited by 1
  2. 2025Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Taskscited by 0
  3. 2025An Efficient Training Pipeline for Reasoning Graphical User Interface Agentscited by 0
  4. 2025V-REX: Benchmarking Exploratory Visual Reasoning via Chain-of-Questionscited by 0
  5. 2026When AI Navigates the Fog of Warcited by 0
  6. 2026An LMM for Precisely Grounding Elements in Documentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ExpVG: Investigating the Design Space of Visual Grounding in Multimodal Large Language Model
  2. 2026SimpleOCR: Rendering Visualized Questions to Teach MLLMs to Read
  3. 2025PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs
  4. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
  5. 2024LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Grounding
  6. 2024The Revolution of Multimodal Large Language Models: A Survey
  7. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  8. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension