mlsanityData to 14 Oct 2026

Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models

arXiv 2505.12343 ↗First posted 18 May 2025cs.LG cs.AI cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucinations, where generated content is inconsistent with the input image. Existing training-free hallucination mitigation methods often suffer from unstable performance and high sensitivity to hyperparameter settings, which limits their practicality and broader adoption. In this paper, we propose Decoding with Inter-layer Consistency via Layer Aggregation (DCLA), a training-free decoding mechanism that requires no retraining, fine-tuning, or access to external knowledge bases. Specifically, DCLA constructs a dynamic semantic reference by aggregating representations from previous layers and uses it to correct semantically deviated layers, thereby enforcing inter-layer consistency. Experiments across seven LVLMs and multiple benchmarks demonstrate the generality of DCLA: it surpasses standard decoding by 28.58 MME points on LLaVA1.5-7B and 42.6 MME points on Qwen2.5-VL, while improving POPE accuracy by 2.74 percentage points in the strongest setting.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 63 of its 79 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

63 of 79 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbations19 Mar
  2. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations3 Oct
  3. 2024Multi-Object Hallucination in Vision-Language Models8 Jul
  4. 2024THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models8 May
  5. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback22 Apr
  6. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  7. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  8. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance13 Feb
  9. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites4 Dec
  10. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  11. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  12. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
6 more
  1. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  2. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  3. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  4. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  5. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  6. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
33 more
  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  5. 2014How transferable are features in deep neural networks?cited by 1,225
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  9. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  10. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  12. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  13. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  14. 2017Deep Layer Aggregationcited by 436
  15. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  16. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  17. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
  18. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  19. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  20. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  21. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 131
  22. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  23. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  24. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Trainingcited by 64
  25. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 59
  26. 2019DIANet: Dense-and-Implicit Attention Networkcited by 37
  27. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 36
  28. 2022Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problemcited by 18
  29. 2024Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEditcited by 14
  30. 2021Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregationcited by 13
  31. 2025From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuningcited by 12
  32. 2023InfMLLM: A Unified Framework for Visual-Language Taskscited by 11
  33. 2025Dynamic Knowledge Integration for Enhanced Vision-Language Reasoningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localizationcited by 4
  2. 2025Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflectioncited by 3
  3. 2026SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineeringcited by 2
  4. 2026Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoningcited by 2
  5. 2025MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processingcited by 1
  6. 2026Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Modelscited by 0
  7. 2026SPARK: Susceptibility-Guided Profiling and Steering of Latent Reasoning States in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
  2. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  3. 2025HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Models
  4. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
  5. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding
  6. 2026Context-Aware Decoding for Faithful Vision-Language Generation
  7. 2026Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance
  8. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning