mlsanityData to 7 Oct 2026

Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey

arXiv 2505.15957 ↗First posted 21 May 2025eess.AS cs.AI cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

With advancements in large audio-language models (LALMs), which enhance large language models (LLMs) with auditory capabilities, these models are expected to demonstrate universal proficiency across various auditory tasks. While numerous benchmarks have emerged to assess LALMs' performance, they remain fragmented and lack a structured taxonomy. To bridge this gap, we conduct a comprehensive survey and propose a systematic taxonomy for LALM evaluations, categorizing them into four dimensions based on their objectives: (1) General Auditory Awareness and Processing, (2) Knowledge and Reasoning, (3) Dialogue-oriented Ability, and (4) Fairness, Safety, and Trustworthiness. We provide detailed overviews within each category and highlight challenges in this field, offering insights into promising future directions. To the best of our knowledge, this is the first survey specifically focused on the evaluations of LALMs, providing clear guidelines for the community. We will release the collection of the surveyed papers and actively maintain it to support ongoing advancements in the field.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 99 of its 160 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

99 of 160 references.

References · 99

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  10. 2024A Survey on LLM-as-a-Judgecited by 1,384
  11. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  12. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
38 more
  1. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  2. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  3. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  4. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  6. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  7. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  8. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  9. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  10. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  11. 2019Clotho: An Audio Captioning Datasetcited by 354
  12. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  13. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  14. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  15. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  16. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 215
  17. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  18. 2016FMA: A Dataset For Music Analysiscited by 205
  19. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 200
  20. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  21. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  22. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  23. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
  24. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 151
  25. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  26. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 118
  27. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  28. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  29. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  30. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
  31. 2020Low-resource Languages: A Review of Past Work and Future Challengescited by 90
  32. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 84
  33. 2024Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queriescited by 78
  34. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  35. 2019A Survey of Code-switched Speech and Language Processingcited by 73
  36. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  37. 2024Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Taskscited by 70
  38. 2023The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluationcited by 68

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 20.

38 papers.

Sort:

By area:

  1. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  2. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 22
  3. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  4. 2025Game-Time: Evaluating Temporal Dynamics in Spoken Language Modelscited by 13
  5. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
  6. 2025Audio Flamingo Sound-CoT Technical Report: Improving Chain-of-Thought Reasoning in Sound Understandingcited by 5
  7. 2025SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Modelscited by 5
  8. 2025Learning Interpretable Features in Audio Latent Spaces via Sparse Autoencoderscited by 5
  9. 2025Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translationcited by 5
  10. 2026Interpretable All-Type Audio Deepfake Detection with Audio LLMs via Frequency-Time Reinforcement Learningcited by 5
  11. 2026All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluationcited by 5
  12. 2025When Silence Matters: The Impact of Irrelevant Audio on Text Reasoning in Large Audio-Language Modelscited by 4
26 more
  1. 2025Hearing the Order: Investigating Position Bias in Large Audio-Language Modelscited by 4
  2. 2025Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variationscited by 4
  3. 2026PRiSM: Benchmarking Phone Realization in Speech Modelscited by 4
  4. 2025AHAMask: Reliable Task Specification for Large Audio Language Models without Instructionscited by 3
  5. 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
  6. 2025TELEVAL: A Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarioscited by 2
  7. 2025Can large audio language models understand child stuttering speech? speech summarization, and source separationcited by 2
  8. 2025AudioMotionBench: Evaluating Auditory Motion Perception in Audio LLMscited by 2
  9. 2026MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Modelscited by 2
  10. 2026Neuron-Level Emotion Control in Speech-Generative Large Audio-Language Modelscited by 2
  11. 2026How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluationcited by 2
  12. 2026VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speechcited by 2
  13. 2026Learning When to Think While Listening in Large Audio-Language Modelscited by 2
  14. 2025Advancing Hearing Assessment: An ASR-Based Frequency-Specific Speech Test for Diagnosing Presbycusiscited by 1
  15. 2026Resurfacing Paralinguistic Awareness in Large Audio Language Modelscited by 1
  16. 2026Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Modelscited by 1
  17. 2026JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructionscited by 1
  18. 2026Synchronization and Turn-Taking in Full-Duplex Speech Dialogue Modelscited by 1
  19. 2025Towards Generalized Source Tracing for Codec-Based Deepfake Speechcited by 0
  20. 2026When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removalcited by 0
  21. 2026SpectCount: Spectrotemporal Counting via Synthetic Signals Improves Large Audio Language Modelscited by 0
  22. 2026Hearing Like Humans? Sound Symbolism and Perceptual Alignment in Speech Language Modelscited by 0
  23. 2026Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Modelscited by 0
  24. 2026Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Modelscited by 0
  25. 2026Caught in the Story: Narrative Captivity in Multi-turn LLMs Conversationcited by 0
  26. 2026MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Modelscited by 0