# GroundSight: Augmenting Vision-Language Models with Grounding Information and De-hallucination

arXiv: https://arxiv.org/abs/2509.25669 · HTML: https://mlsanity.com/paper/2509.25669/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)
- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 10 of 10)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [Fusion of Detected Objects in Text for Visual Question Answering](https://mlsanity.com/paper/1908.05054/) (2019)
- [Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models](https://mlsanity.com/paper/2403.12966/) (2024)
- [Scene-Text Grounding for Text-Based Video Question Answering](https://mlsanity.com/paper/2409.14319/) (2024)
- [Locate Then Generate: Bridging Vision and Language with Bounding Box for Scene-Text VQA](https://mlsanity.com/paper/2304.01603/) (2023)

## Built on by (most cited first, 1 of 1)

- [SphMind: Towards Robust, Training-Free VLM-based Spatial Reasoning with a 360 Camera](https://mlsanity.com/paper/2609.33462/) (2026)

## Abstract (arXiv)

We propose a method to improve Visual Question Answering (VQA) with Retrieval-Augmented Generation (RAG) by introducing text-grounded object localization. Rather than retrieving information based on the entire image, our approach enables the model to generate a bounding box around the object most relevant to the question, allowing for targeted image cropping and focused retrieval. This reduces background noise, improves alignment between visual and textual cues, and helps mitigate hallucinations. Our RAG method enhances context-aware VQA responses increased the accuracy from 22.19% to 25.64%, with an absolute increase of 3.45 percentage points, compared to the baseline Llama-3.2-Vision-11B agent. We also proposed a de-hallucination method based on question type which can effectively reduce the hallucination rate from 65.79% to 13.88% and improves the truthfulness score.