mlsanityData to 7 Oct 2026

Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection

arXiv 2509.03113 ↗First posted 3 Sep 2025cs.CV cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal large language models achieve strong performance across diverse tasks but remain prone to hallucinations, where outputs are not grounded in visual inputs. This issue can be attributed to two main biases: text-visual bias, the overreliance on prompts and prior outputs, and co-occurrence bias, spurious correlations between frequently paired objects. We propose Gradient-based Influence-Aware Constrained Decoding (GACD), an inference-based method, that addresses both biases without auxiliary models, and is readily applicable to existing models without finetuning. The core of our approach is bias estimation, which uses first-order Taylor gradients to understand the contribution of individual tokens-visual features and text tokens-to the current output. Based on this analysis, GACD mitigates hallucinations through two components: (1) suppressing spurious visual features correlated with the output objects, and (2) rebalancing cross-modal contributions by strengthening visual features relative to text. Experiments across multiple benchmarks demonstrate that GACD effectively reduces hallucinations and improves the visual grounding of MLLM outputs.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 63 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

51 of 63 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025Mitigating Object Hallucinations via Sentence-Level Early Intervention16 Jul
  2. 2025LPOI: Listwise Preference Optimization for Vision Language Models27 May
  3. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens23 Nov
  4. 2024CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs19 Nov
  5. 2024ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models25 Aug
  6. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)6 Aug
  7. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  8. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models28 May
  9. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  10. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  11. 2024EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models15 Feb
  12. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
7 more
  1. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  3. 2023Mitigating Hallucination in Visual Language Models with Visual Supervision27 Nov
  4. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  5. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Models3 Oct
  6. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  7. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  11. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
20 more
  1. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  4. 2018Object Hallucination in Image Captioningcited by 661
  5. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  6. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  7. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  9. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  10. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  11. 2018Learning Not to Learn: Training Deep Neural Networks with Biased Datacited by 255
  12. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  13. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 168
  14. 2021Guided Integrated Gradients: An Adaptive Path Method for Removing Noisecited by 102
  15. 2024Bias in Large Language Models: Origin, Evaluation, and Mitigationcited by 77
  16. 2022A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributionscited by 64
  17. 2023Discovering and Mitigating Visual Biases through Keyword Explanationcited by 58
  18. 2023Sequential Integrated Gradients: a simple but effective method for explaining language modelscited by 52
  19. 2023Impact of Co-occurrence on Factual Knowledge of Large Language Modelscited by 19
  20. 2023A Solution to Co-occurrence Bias: Attributes Disentanglement via Mutual Information Minimization for Pedestrian Attribute Recognitioncited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 19
  2. 2026SAVOR: Self-Aware Visual Grounding via Confidence-Calibrated Reinforcement Learning for Multimodal Hallucination Mitigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  2. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
  3. 2025Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
  4. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  5. 2025Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images
  6. 2025Preemptive Hallucination Reduction: An Input-Level Approach for Multimodal Language Model
  7. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation
  8. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence