# Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decoding

arXiv: https://arxiv.org/abs/2506.07233 · HTML: https://mlsanity.com/paper/2506.07233/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 39 references that are in the map.
- First posted on arXiv 8 Jun 2025.

## Builds on (most cited first, 30 of 36)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [A Survey of Hallucination in Large Foundation Models](https://mlsanity.com/paper/2309.05922/) (2023)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://mlsanity.com/paper/2305.14739/) (2023)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [MM-LLMs: Recent Advances in MultiModal Large Language Models](https://mlsanity.com/paper/2401.13601/) (2024)
- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [The Revolution of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2402.12451/) (2024)
- [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://mlsanity.com/paper/2503.03983/) (2025)
- [AudioBench: A Universal Benchmark for Audio Large Language Models](https://mlsanity.com/paper/2406.16020/) (2024)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech](https://mlsanity.com/paper/2309.09510/) (2023)
- [Joint Audio and Speech Understanding](https://mlsanity.com/paper/2309.14405/) (2023)
- [Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering](https://mlsanity.com/paper/2204.09634/) (2022)
- [Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks](https://mlsanity.com/paper/2411.05361/) (2024)
- [DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data](https://mlsanity.com/paper/2409.20007/) (2024)
- [SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information](https://mlsanity.com/paper/2505.13237/) (2025)
- [Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2505.15957/) (2025)

## Built on by (most cited first, 24 of 24)

- [Silence is Golden: Mitigating Hallucinations in Large Audio-Language Models via Layer-Weighted Vector Steering](https://mlsanity.com/paper/2510.12851/) (2025)
- [How Contrastive Decoding Enhances Large Audio Language Models](https://mlsanity.com/paper/2603.09232/) (2026)
- [ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Models](https://mlsanity.com/paper/2604.10065/) (2026)
- [Temporal Contrastive Decoding: A Training-Free Method for Large Audio-Language Models](https://mlsanity.com/paper/2604.15383/) (2026)
- [Beyond Text Following: Repairable Arbitration Reversals in Audio-Language Models](https://mlsanity.com/paper/2606.05161/) (2026)
- [CAAD: Contrastive Audio-Aware Distillation for Efficient Speech Language Models](https://mlsanity.com/paper/2606.23052/) (2026)
- [Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Time](https://mlsanity.com/paper/2605.01766/) (2026)
- [Adaptive Perturbation Selection for Contrastive Audio Decoding](https://mlsanity.com/paper/2607.00247/) (2026)
- [TAD: Token-Adaptive Contrastive Decoding with Confidence-Guided Gating for Hallucination Mitigation in Large Audio-Language Models](https://mlsanity.com/paper/2609.07286/) (2026)
- [Evaluating Hallucinations in Audio-Visual Multimodal LLMs with Spoken Queries under Diverse Acoustic Conditions](https://mlsanity.com/paper/2510.08581/) (2025)
- [CAF-Score: Calibrating CLAP with LALMs for Reference-free Audio Captioning Evaluation](https://mlsanity.com/paper/2603.19615/) (2026)
- [Self-Correction Can Amplify Hallucinations: Fact-Level Repair with Graph-Based Evidence Routing in Multimodal Generation](https://mlsanity.com/paper/2606.00232/) (2026)
- [Why Sampling Is Not Choosing: Intentionality, Agency, and Moral Responsibility in Large Language Models](https://mlsanity.com/paper/2606.13441/) (2026)
- [Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Models](https://mlsanity.com/paper/2607.11801/) (2026)
- [Attention-Guided Reliability Scaling for Contrastive Decoding in Robust Audio-Visual Speech Recognition](https://mlsanity.com/paper/2608.26213/) (2026)
- [Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions](https://mlsanity.com/paper/2609.02940/) (2026)
- [Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts](https://mlsanity.com/paper/2609.06011/) (2026)
- [Music Hallucination in Audio-Language Models: A Hierarchical Formulation and Empirical Study](https://mlsanity.com/paper/2609.20195/) (2026)
- [REVE: Efficient Hallucination Correction for Large Audio-Language Models via Reused Encoder States](https://mlsanity.com/paper/2609.26028/) (2026)
- [Reward-Tilted On-Policy Distillation for Acoustic Grounding in Audio-Language Models](https://mlsanity.com/paper/2609.28778/) (2026)
- [Broadening Uncertainty Estimation for Audio Question Answering Across Methods, Formats, and Inputs](https://mlsanity.com/paper/2609.28879/) (2026)
- [MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models](https://mlsanity.com/paper/2609.33893/) (2026)
- [AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models](https://mlsanity.com/paper/2610.00706/) (2026)
- [Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination](https://mlsanity.com/paper/2610.13112/) (2026)

## Abstract (arXiv)

Large Audio-Language Models (LALMs) can take audio and text as the inputs and answer questions about the audio. While prior LALMs have shown strong performance on standard benchmarks, there has been alarming evidence that LALMs can hallucinate what is presented in the audio. To mitigate the hallucination of LALMs, we introduce Audio-Aware Decoding (AAD), a lightweight inference-time strategy that uses contrastive decoding to compare the token prediction logits with and without the audio context. By contrastive decoding, AAD promotes the tokens whose probability increases when the audio is present. We conduct our experiment on object hallucination datasets with three LALMs and show that AAD improves the F1 score by 0.046 to 0.428. We also show that AAD can improve the accuracy on general audio QA datasets like Clotho-AQA by 5.4% to 10.3%. We conduct thorough ablation studies to understand the effectiveness of each component in AAD.