mlsanityData to 7 Oct 2026

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

arXiv 2601.21181 ↗First posted 29 Jan 2026cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a training-free method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model's inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8% and 2.0% improvements for VideoLLaMA2-AV, 8.7% and 4.7% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods. Our code is available at hrefhttps://github.com/top-yun/MADhttps://github.com/top-yun/MAD

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 50 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

46 of 50 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization4 Jun
  2. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding27 May
  3. 2025Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding23 May
  4. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection18 Dec
  5. 2024Mitigating Object Hallucination via Concentric Causal Attention21 Oct
  6. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio16 Oct
  7. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation15 Oct
  8. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  9. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models4 Jun
  10. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  11. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  12. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
5 more
  1. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  2. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  3. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  4. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  5. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  9. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  10. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  11. 2025Qwen2.5-Omni Technical Reportcited by 704
  12. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
17 more
  1. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  2. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  3. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  4. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  5. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  6. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  7. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  8. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  9. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  10. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  11. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  12. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  13. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  14. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  15. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  16. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 49
  17. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMscited by 0
  2. 2026Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization
  2. 2025When Images Speak Louder: Mitigating Language Bias-induced Hallucinations in VLMs through Cross-Modal Guidance
  3. 2025Robust Multimodal Large Language Models Against Modality Conflict
  4. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigation
  5. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality
  6. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  7. 2025Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection
  8. 2026Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Time