mlsanityData to 14 Oct 2026

Hallucination-aware intermediate representation edit in large vision-language models

arXiv 2603.29405 ↗First posted 31 Mar 2026cs.CV cs.AIcited by 0

On the map: LLM reasoning → Hallucination detection and mitigation

also in 1 more
Abstract (arXiv)

Large Vision-Language Models have demonstrated exceptional performance in multimodal reasoning and complex scene understanding. However, these models still face significant hallucination issues, where outputs contradict visual facts. Recent research on hallucination mitigation has focused on retraining methods and Contrastive Decoding (CD) methods. While both methods perform well, retraining methods require substantial training resources, and CD methods introduce dual inference overhead. These factors hinder their practical applicability. To address the above issue, we propose a framework for dynamically detecting hallucination representations and performing hallucination-eliminating edits on these representations. With minimal additional computational cost, we achieve state-of-the-art performance on existing benchmarks. Extensive experiments demonstrate the effectiveness of our approach, highlighting its efficient and robust hallucination elimination capability and its powerful controllability over hallucinations. Code is available at https://github.com/ASGO-MM/HIRE

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning. Builds on 54 of its 63 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

54 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models28 Mar
  2. 2024Mitigating Object Hallucination via Concentric Causal Attention21 Oct

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2025Qwen2.5-VL Technical Reportcited by 5,587
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  12. 2023Qwen Technical Reportcited by 3,344
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  8. 2023A Survey on Multimodal Large Language Modelscited by 909
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  10. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  11. 2018Object Hallucination in Image Captioningcited by 669
  12. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  13. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  14. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  15. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  16. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  17. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  18. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 225
  19. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 214
  20. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  21. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  22. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  23. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 121
  24. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 104
  25. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 94
  26. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 91
  27. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 47
  28. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  29. 2023Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuningcited by 45
  30. 2024A Survey on Large Language Model Hallucination via a Creativity Perspectivecited by 38
  31. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decodingcited by 24
  32. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 23
  33. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)cited by 23
  34. 2024Reference-free Hallucination Detection for Large Vision-Language Modelscited by 23
  35. 2023Is Bigger and Deeper Always Better? Probing LLaMA Across Scales and Layerscited by 20
  36. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimizationcited by 20
  37. 2023Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Modelscited by 13
  38. 2025Towards Understanding How Knowledge Evolves in Large Vision-Language Modelscited by 10

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
  2. 2026SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
  3. 2025Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchors
  4. 2026CHASD: Language Increment-Calibrated Contrastive Decoding against Hallucination in LVLMs
  5. 2026Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation
  6. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  7. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning
  8. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models