mlsanityData to 7 Oct 2026

Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models

arXiv 2406.08402 ↗First posted 12 Jun 2024eess.AS cs.CL cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Large audio-language models (LALMs) enhance traditional large language models by integrating audio perception capabilities, allowing them to tackle audio-related tasks. Previous research has primarily focused on assessing the performance of LALMs across various tasks, yet overlooking their reliability, particularly concerning issues like object hallucination. In our study, we introduce methods to assess the extent of object hallucination of publicly available LALMs. Our findings reveal that LALMs are comparable to specialized audio captioning models in their understanding of audio content, but struggle to answer discriminative questions, specifically those requiring the identification of the presence of particular object sounds within an audio clip. This limitation highlights a critical weakness in current LALMs: their inadequate understanding of discriminative queries. Moreover, we explore the potential of prompt engineering to enhance LALMs' performance on discriminative questions.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 33 of its 37 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

33 of 37 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  6. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  7. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  8. 2018Object Hallucination in Image Captioningcited by 661
  9. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  10. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  12. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
21 more
  1. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  2. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  3. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  4. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  5. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  6. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  7. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  8. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  9. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  10. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  11. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  12. 2023Joint Audio and Speech Understandingcited by 99
  13. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  14. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  15. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  16. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  17. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  18. 2023LLaSM: Large Language and Speech Modelcited by 56
  19. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  20. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 30
  21. 2023A Whisper transformer for audio captioning trained with synthetic captions and transfer learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 11; 2026: 17.

30 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  2. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  3. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  4. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  5. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  6. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 22
  7. 2025Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samplescited by 11
  8. 2025Aligned Better, Listen Better for Audio-Visual Large Language Modelscited by 10
  9. 2025Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interactioncited by 9
  10. 2026AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answeringcited by 8
  11. 2025CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Modelscited by 7
  12. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
18 more
  1. 2025BRACE: A Benchmark for Robust Audio Caption Quality Evaluationcited by 7
  2. 2025Silence is Golden: Mitigating Hallucinations in Large Audio-Language Models via Layer-Weighted Vector Steeringcited by 6
  3. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  4. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  5. 2026Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Timecited by 1
  6. 2026Steering Where to Listen: Instruction-Based Activation Steering Redirects Temporal Attention in Large Audio-Language Modelscited by 1
  7. 2026Adaptive Perturbation Selection for Contrastive Audio Decodingcited by 1
  8. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  9. 2026Localizing and Editing Knowledge in Large Audio-Language Modelscited by 0
  10. 2026Multi-Source Evidence Fusion for Audio Question Answeringcited by 0
  11. 2026Noise-Aware In-Context Learning for Hallucination Mitigation in ALLMscited by 0
  12. 2026Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognitioncited by 0
  13. 2026AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Modelscited by 0
  14. 2026Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Modelscited by 0
  15. 2026Tracing Audio Grounding and Answer Selection in Audio LLMscited by 0
  16. 2026TAD: Token-Adaptive Contrastive Decoding with Confidence-Guided Gating for Hallucination Mitigation in Large Audio-Language Modelscited by 0
  17. 2026Broadening Uncertainty Estimation for Audio Question Answering Across Methods, Formats, and Inputscited by 0
  18. 2026MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Modelscited by 0