mlsanityData to 7 Oct 2026

Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Models

arXiv 2309.04041 ↗First posted 7 Sep 2023cs.CV cs.CLcited by 13

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Vision-Language Models (LVLMs) offer remarkable benefits for a variety of vision-language tasks. However, a challenge hindering their application in real-world scenarios, particularly regarding safety, robustness, and reliability, is their constrained semantic grounding ability, which pertains to connecting language to the physical-world entities or concepts referenced in images. Therefore, a crucial need arises for a comprehensive study to assess the semantic grounding ability of widely used LVLMs. Despite the significance, sufficient investigation in this direction is currently lacking. Our work bridges this gap by designing a pipeline for generating large-scale evaluation datasets covering fine-grained semantic information, such as color, number, material, etc., along with a thorough assessment of seven popular LVLMs' semantic grounding ability. Results highlight prevalent misgrounding across various aspects and degrees. To address this issue, we propose a data-centric enhancement method that aims to improve LVLMs' semantic grounding ability through multimodal instruction tuning on fine-grained conversations. Experiments on enhanced LVLMs demonstrate notable improvements in addressing misgrounding issues.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 42 references that are in the map. First posted on arXiv 7 Sep 2023.

Builds on

27 of 42 references.

References · 27

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2021Grounded Language-Image Pre-trainingcited by 1,147
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
15 more
  1. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  2. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  3. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 411
  4. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  5. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  6. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  7. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  8. 2023PACO: Parts and Attributes of Common Objectscited by 146
  9. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  10. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
  11. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  12. 2019SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognitioncited by 55
  13. 2021Does Vision-and-Language Pretraining Improve Lexical Grounding?cited by 25
  14. 2023Tackling Vision Language Tasks Through Learning Inner Monologuescited by 6
  15. 2023MuG: A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fieldscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 5; 2026: 2.

13 papers.

Sort:

By area:

  1. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  2. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 51
  3. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 46
  4. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decodingcited by 23
  5. 2023Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Modelscited by 12
  6. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decodingcited by 12
  7. 2025Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decodingcited by 6
  8. 2025Mitigating Low-Level Visual Hallucinations Requires Self-Awareness: Database, Model and Training Strategycited by 1
  9. 2026ProGAL-VLA: Grounded Alignment through Prospective Reasoning in Vision-Language-Action Modelscited by 1
  10. 2024Hybrid Primal Sketch: Combining Analogy, Qualitative Representations, and Computer Vision for Scene Understandingcited by 0
  11. 2025PictSure: Pretraining Embeddings Matters for In-Context Learning Image Classifierscited by 0
  12. 2025Q-Doc: Benchmarking Document Image Quality Assessment Capabilities in Multi-modal Large Language Modelscited by 0
1 more
  1. 2026Hallucination-aware intermediate representation edit in large vision-language modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?
  2. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
  3. 2025Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Grounding
  4. 2025Towards Understanding Visual Grounding in Visual Language Models
  5. 2026When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs
  6. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
  7. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  8. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding