mlsanityData to 14 Oct 2026

MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation

arXiv 2410.11779 ↗First posted 15 Oct 2024cs.CL cs.AI cs.CVcited by 88

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) frequently exhibit hallucination phenomena, but the underlying reasons remain poorly understood. In this paper, we present an empirical analysis and find that, although MLLMs incorrectly generate the objects in the final output, they are actually able to recognize visual objects in the preceding layers. We speculate that this may be due to the strong knowledge priors of the language model suppressing the visual information, leading to hallucinations. Motivated by this, we propose a novel dynamic correction decoding method for MLLMs DeCo, which adaptively selects the appropriate preceding layers and proportionally integrates knowledge into the final layer to adjust the output logits. Note that DeCo is model agnostic and can be seamlessly incorporated with various classic decoding strategies and applied to different MLLMs. We evaluate DeCo on widely-used benchmarks, demonstrating that it can reduce hallucination rates by a large margin compared to baselines, highlighting its potential to mitigate hallucinations. Code is available at https://github.com/zjunlp/DeCo.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 59 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

53 of 59 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  2. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  3. 2024Unified Hallucination Detection for Multimodal Large Language Models5 Feb
  4. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  5. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  8. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  9. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  10. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  11. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  12. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
1 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 40

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  9. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  10. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  12. 2018Object Hallucination in Image Captioningcited by 669
28 more
  1. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  2. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  3. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  5. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  6. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  7. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  8. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  9. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
  10. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 323
  11. 2022Confident Adaptive Language Modelingcited by 310
  12. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  13. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  14. 2019Depth-Adaptive Transformercited by 236
  15. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  16. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  17. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 200
  18. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  19. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 114
  20. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 108
  21. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  22. 2024In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigationcited by 45
  23. 2024Debiasing Multimodal Large Language Models via Penalization of Language Priorscited by 37
  24. 2024Insights into LLM Long-Context Failures: When Transformers Know but Don't Tellcited by 36
  25. 2023FactCHD: Benchmarking Fact-Conflicting Hallucination Detectioncited by 29
  26. 2023Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPTcited by 28
  27. 2024Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Modelscited by 25
  28. 2024Llama SLayer 8B: Shallow Layers Hold the Key to Knowledge Injectioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 38; 2026: 49.

88 papers.

Sort:

By area:

  1. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 50
  2. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  3. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMscited by 17
  4. 2025When Thinking Drifts: Evidential Grounding for Robust Video Reasoningcited by 16
  5. 2025See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Modelcited by 15
  6. 2025RSCC: A Large-Scale Remote Sensing Change Caption Dataset for Disaster Eventscited by 14
  7. 2025Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flowcited by 13
  8. 2025Fast Large Language Model Collaborative Decoding via Speculationcited by 12
  9. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  10. 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lenscited by 8
  11. 2026Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Modelscited by 8
  12. 2024Systematic Reward Gap Optimization for Mitigating VLM Hallucinationscited by 7
38 more
  1. 2025ActionArt: Advancing Multimodal Large Models for Fine-Grained Human-Centric Video Understandingcited by 7
  2. 2025Direct Visual Grounding by Directing Attention of Visual Tokenscited by 7
  3. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 7
  4. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMscited by 7
  5. 2025CoFFT: Chain of Foresight-Focus Thought for Visual Language Modelscited by 6
  6. 2026Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMscited by 6
  7. 2025Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Modelscited by 5
  8. 2025PostAlign: Multimodal Grounding as a Corrective Lens for MLLMscited by 5
  9. 2025On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Modelscited by 5
  10. 2025Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMscited by 5
  11. 2025Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigationcited by 5
  12. 2025SAVE: Sparse Autoencoder-Driven Visual Information Enhancement for Mitigating Object Hallucinationcited by 5
  13. 2026MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Modelscited by 5
  14. 2025HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Modelscited by 4
  15. 2025Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localizationcited by 4
  16. 2025Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchorscited by 4
  17. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4
  18. 2025Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Modelscited by 4
  19. 2025Towards Reliable LLM-based Robot Planning via Combined Uncertainty Estimationcited by 4
  20. 2026Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMscited by 4
  21. 2026FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMscited by 4
  22. 2026Mitigating Manifold Departure: Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decodingcited by 4
  23. 2026Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucinationcited by 4
  24. 2025PruneHal: Reducing Hallucinations in Multi-modal Large Language Models through Adaptive KV Cache Pruningcited by 3
  25. 2025CLASH: A Benchmark for Cross-Modal Contradiction Detectioncited by 3
  26. 2025Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignmentcited by 3
  27. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steeringcited by 3
  28. 2026RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrievalcited by 3
  29. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flowcited by 3
  30. 2025SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revisioncited by 2
  31. 2026EAGLE: Expert-Augmented Attention Guidance for Tuning-Free Industrial Anomaly Detection in Multimodal Large Language Modelscited by 2
  32. 2026Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplerscited by 2
  33. 2026HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Modelscited by 2
  34. 2026AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLMcited by 2
  35. 2026SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigationcited by 2
  36. 2026Mitigating Multimodal Hallucination via Phase-wise Self-rewardcited by 2
  37. 2026When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMscited by 2
  38. 2026MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialoguecited by 2

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?
  2. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
  3. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
  4. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  5. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding
  6. 2025Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decoding
  7. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  8. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMs