mlsanityData to 7 Oct 2026

ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM

arXiv 2506.14766 ↗First posted 17 Jun 2025cs.CV cs.CLcited by 36

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal large language models (MLLMs) frequently hallucinate by over-committing to spurious visual cues. Prior remedies-Visual and Instruction Contrastive Decoding (VCD, ICD)-mitigate this issue, yet the mechanism remains opaque. We first empirically show that their improvements systematically coincide with redistributions of cross-modal attention. Building on this insight, we propose Attention-Steerable Contrastive Decoding (ASCD), which directly steers the attention scores during decoding. ASCD combines (i) positive steering, which amplifies automatically mined text-centric heads-stable within a model and robust across domains-with (ii) negative steering, which dampens on-the-fly identified critical visual tokens. The method incurs negligible runtime and memory overhead and requires no additional training. Across five MLLM backbones and three decoding schemes, ASCD reduces hallucination on POPE, CHAIR, and MMHal-Bench by up to 38.2 percent while improving accuracy on standard VQA benchmarks, including MMMU, MM-VET, ScienceQA, TextVQA, and GQA. These results position attention steering as a simple, model-agnostic, and principled route to safer, more faithful multimodal generation.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 69 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

44 of 69 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  2. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  3. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  4. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  6. 2023Mitigating Hallucination in Visual Language Models with Visual Supervision27 Nov
  7. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct

Other references · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
25 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  2. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  4. 2018Object Hallucination in Image Captioningcited by 661
  5. 2016Visual Dialogcited by 377
  6. 2023How Language Model Hallucinations Can Snowballcited by 310
  7. 2024MACE: Mass Concept Erasure in Diffusion Modelscited by 298
  8. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  9. 2021AdaRNN: Adaptive Learning and Forecasting of Time Seriescited by 120
  10. 2024Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advancescited by 105
  11. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 88
  12. 2024ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guaranteescited by 62
  13. 2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selectioncited by 57
  14. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 50
  15. 2025CoT-Kinetics: A Theoretical Modeling Assessing LRM Reasoning Processcited by 50
  16. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 34
  17. 2025AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineagecited by 29
  18. 2025ChartEdit: How Far Are MLLMs From Automating Chart Analysis? Evaluating MLLMs' Capability via Chart Editingcited by 28
  19. 2025MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledgecited by 22
  20. 2025Self-Evolving Multi-Agent Systems via Textual Backpropagationcited by 21
  21. 2023SPOT! Revisiting Video-Language Models for Event Understandingcited by 20
  22. 2025Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generationcited by 14
  23. 2024Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-upcited by 10
  24. 2023Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Modelscited by 9
  25. 2025PreP-OCR: A Complete Pipeline for Document Image Restoration and Enhanced OCR Accuracycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 26.

36 papers.

Sort:

By area:

  1. 2026ReTrack: Evidence-Driven Dual-Stream Directional Anchor Calibration Network for Composed Video Retrievalcited by 39
  2. 2025HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrievalcited by 32
  3. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 31
  4. 2026INTENT: Invariance and Discrimination-aware Noise Mitigation for Robust Composed Image Retrievalcited by 26
  5. 2026HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrievalcited by 24
  6. 2026Air-Know: Arbiter-Calibrated Knowledge-Internalizing Robust Network for Composed Image Retrievalcited by 17
  7. 2026ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrievalcited by 15
  8. 2026TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrievalcited by 13
  9. 2026Dual Latent Memory for Visual Multi-agent Systemcited by 9
  10. 2026OPD-V: Visual On-Policy Self-Distillation with Modality Balancecited by 7
  11. 2025Improving Multimodal Sentiment Analysis via Modality Optimization and Dynamic Primary Modality Selectioncited by 6
  12. 2025TECP: Token-Entropy Conformal Prediction for LLMscited by 5
24 more
  1. 2025Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localizationcited by 3
  2. 2026Set-Valued Prediction for Large Language Models with Feasibility-Aware Coverage Guaranteescited by 3
  3. 2026EchoRL: Reinforcement Learning via Rollout Echoingcited by 3
  4. 2026MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialoguecited by 2
  5. 2026ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoningcited by 2
  6. 2025Conformal Sets in Multiple-Choice Question Answering under Black-Box Settings with Provable Coverage Guaranteescited by 1
  7. 2026When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMscited by 1
  8. 2026Causal Evidence for Attention Head Imbalance in Modality Conflict Hallucinationcited by 1
  9. 2026Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guaranteescited by 1
  10. 2026Inference-Time Attention Steering for Vision-Language-Action Driving Modelscited by 1
  11. 2025FADE: Adversarial Concept Erasure in Flow Modelscited by 0
  12. 2025Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Controlcited by 0
  13. 2025Robust Concept Erasure in Diffusion Models: A Theoretical Perspective on Security and Robustnesscited by 0
  14. 2025KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controlscited by 0
  15. 2026When Denoising Becomes Unsigning: Theoretical and Empirical Analysis of Watermark Fragility Under Diffusion-Based Image Editingcited by 0
  16. 2026Editing Away the Evidence: Diffusion-Based Image Manipulation and the Failure Modes of Robust Watermarkingcited by 0
  17. 2026Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchorscited by 0
  18. 2026See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignmentcited by 0
  19. 2026State Beyond Appearance: Diagnosing and Improving State Consistency in Dial-Based Measurement Readingcited by 0
  20. 2026GazeVLM: Active Vision via Internal Attention Control for Multimodal Reasoningcited by 0
  21. 2026Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigationcited by 0
  22. 2026MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Biascited by 0
  23. 2026Spectral Query-Key Product Weight Steering for Training-Free VLM Hallucination Mitigationcited by 0
  24. 2026Attending to Multimodal Generation One Token at a Timecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation
  2. 2024Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding
  3. 2026Rethinking CD: A Reproducibility Study and Extension on the Ineffectiveness of Contrastive Decoding at Mitigating Object Hallucinations in MLLMs
  4. 2025MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding
  5. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding
  6. 2025Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding
  7. 2025The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?
  8. 2026Mask What Matters: Mitigating Object Hallucinations in Multimodal Large Language Models with Object-Aligned Visual Contrastive Decoding