mlsanityData to 14 Oct 2026

Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning

arXiv 2410.16130 ↗First posted 21 Oct 2024eess.AS cs.CL cs.SDcited by 31

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Recent advancements in large audio-language models (LALMs) have shown impressive capabilities in understanding and reasoning about audio and speech information. However, these models still face challenges, including hallucinating non-existent sound events, misidentifying the order of sound events, and incorrectly attributing sound sources, which undermine their reliability and real-world application. To systematically evaluate these issues, we propose three distinct tasks: object existence, temporal order, and object attribute within audio. These tasks assess the models' comprehension of critical audio information aspects. Our experimental results reveal limitations in these fundamental tasks, underscoring the need for better models in recognizing specific sound events, determining event sequences, and identifying sound sources. To improve performance in these areas, we introduce a multi-turn chain-of-thought approach, which demonstrates significantly improved model performance across the proposed tasks.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 45 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

41 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities17 Jun
  2. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs12 Nov
  3. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models12 Oct

Other references · 38

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  6. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  7. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  8. 2024Qwen2-Audio Technical Reportcited by 560
  9. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  10. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  11. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  12. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
26 more
  1. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  2. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  3. 2023Pengi: An Audio Language Model for Audio Taskscited by 220
  4. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  5. 2023Listen, Think, and Understandcited by 209
  6. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  7. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  8. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  9. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  10. 2023Joint Audio and Speech Understandingcited by 100
  11. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  12. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  13. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  14. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  15. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 72
  16. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  17. 2022Vocalsound: A Dataset for Improving Human Vocal Sounds Recognitioncited by 62
  18. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 44
  19. 2023Chainpoll: A high efficacy method for LLM hallucination detectioncited by 39
  20. 2024BLSP-Emo: Towards Empathetic Large Speech-Language Modelscited by 36
  21. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 33
  22. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 31
  23. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 31
  24. 2024EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioningcited by 28
  25. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  26. 2024BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Modelscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 21.

31 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  2. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 39
  3. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 24
  4. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  5. 2025Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samplescited by 11
  6. 2025Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoningcited by 10
  7. 2026AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answeringcited by 8
  8. 2026HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Modelscited by 8
  9. 2025BRACE: A Benchmark for Robust Audio Caption Quality Evaluationcited by 7
  10. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  11. 2025FESTA: Functionally Equivalent Sampling for Trust Assessment of Multimodal LLMscited by 5
  12. 2026Causal Tracing of Audio-Text Fusion in Large Audio Language Modelscited by 5
19 more
  1. 2026Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steeringcited by 4
  2. 2026Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Modelscited by 4
  3. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  4. 2026Encode Once, Decode Never: Reusing Audio LM Internals for Efficient Temporal Localizationcited by 2
  5. 2025Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updatescited by 1
  6. 2026Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Timecited by 1
  7. 2026Adaptive Perturbation Selection for Contrastive Audio Decodingcited by 1
  8. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  9. 2026Tracing Audio Grounding and Answer Selection in Audio LLMscited by 1
  10. 2025ColLab: A Collaborative Spatial Progressive Data Engine for Referring Expression Comprehension and Generationcited by 0
  11. 2026Localizing and Editing Knowledge in Large Audio-Language Modelscited by 0
  12. 2026When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removalcited by 0
  13. 2026Back to Basics: Revisiting ASR in the Age of Voice Agentscited by 0
  14. 2026Noise-Aware In-Context Learning for Hallucination Mitigation in ALLMscited by 0
  15. 2026Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Modelscited by 0
  16. 2026Music Hallucination in Audio-Language Models: A Hierarchical Formulation and Empirical Studycited by 0
  17. 2026MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Modelscited by 0
  18. 2026AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Modelscited by 0
  19. 2026Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucinationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Investigating Faithfulness in Large Audio Language Models
  2. 2026A Closer Look at Failure Modes in Temporal Understanding of Large Audio-Language Models
  3. 2025Thinking with Sound: Audio Chain-of-Thought Enables Multimodal Reasoning in Large Audio-Language Models
  4. 2025AHA: Aligning Large Audio-Language Models for Reasoning Hallucinations via Counterfactual Hard Negatives
  5. 2026Continuous Audio Thinking for Large Audio Language Models
  6. 2025Towards Reliable Large Audio Language Model
  7. 2026MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models
  8. 2025Scaling Auditory Cognition via Test-Time Compute in Audio Language Models