mlsanityData to 14 Oct 2026

Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples

arXiv 2505.14518 ↗First posted 20 May 2025eess.AS cs.CL cs.SDcited by 11

Not on the map. Why?

Abstract (arXiv)

Recent advancements in audio-aware large language models (ALLMs) enable them to process and understand audio inputs. However, these models often hallucinate non-existent sound events, reducing their reliability in real-world applications. To address this, we propose LISTEN (Learning to Identify Sounds Through Extended Negative Samples), a contrastive-like training method that enhances ALLMs' ability to distinguish between present and absent sounds using synthesized data from the backbone LLM. Unlike prior approaches, our method requires no modification to LLM parameters and efficiently integrates audio representations via a lightweight adapter. Experiments show that LISTEN effectively mitigates hallucinations while maintaining impressive performance on existing audio question and reasoning benchmarks. At the same time, it is more efficient in both data and computation.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

29 of 37 references.

References · 29

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  9. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  10. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  11. 2024Qwen2-Audio Technical Reportcited by 560
  12. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
17 more
  1. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  2. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  3. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  4. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  5. 2023Joint Audio and Speech Understandingcited by 100
  6. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 85
  7. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  8. 2024Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Taskscited by 71
  9. 2022Vocalsound: A Dataset for Improving Human Vocal Sounds Recognitioncited by 62
  10. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 45
  11. 2023Chainpoll: A high efficacy method for LLM hallucination detectioncited by 39
  12. 2024BLSP-Emo: Towards Empathetic Large Speech-Language Modelscited by 36
  13. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 33
  14. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 31
  15. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 31
  16. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  17. 2023Investigating the Emergent Audio Classification Ability of ASR Foundation Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answeringcited by 8
  2. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
  3. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  4. 2026How Contrastive Decoding Enhances Large Audio Language Modelscited by 6
  5. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  6. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  7. 2026TAD: Token-Adaptive Contrastive Decoding with Confidence-Guided Gating for Hallucination Mitigation in Large Audio-Language Modelscited by 1
  8. 2026SpectCount: Spectrotemporal Counting via Synthetic Signals Improves Large Audio Language Modelscited by 0
  9. 2026Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognitioncited by 0
  10. 2026Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Modelscited by 0
  11. 2026AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Modelscited by 0