mlsanityData to 7 Oct 2026

AudioBench: A Universal Benchmark for Audio Large Language Models

arXiv 2406.16020 ↗First posted 23 Jun 2024cs.SD cs.CL eess.AScited by 124

Not on the map. Why?

Cited by 27 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce AudioBench, a universal benchmark designed to evaluate Audio Large Language Models (AudioLLMs). It encompasses 8 distinct tasks and 26 datasets, among which, 7 are newly proposed datasets. The evaluation targets three main aspects: speech understanding, audio scene understanding, and voice understanding (paralinguistic). Despite recent advancements, there lacks a comprehensive benchmark for AudioLLMs on instruction following capabilities conditioned on audio signals. AudioBench addresses this gap by setting up datasets as well as desired evaluation metrics. Besides, we also evaluated the capabilities of five popular models and found that no single model excels consistently across all tasks. We outline the research outlook for AudioLLMs and anticipate that our open-sourced evaluation toolkit, data, and leaderboard will offer a robust testbed for future model developments.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Builds on 50 of its 78 references that are in the map. First posted on arXiv 23 Jun 2024.

Builds on

50 of 78 references.

References · 50

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2023Mistral 7Bcited by 3,065
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  12. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
38 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  3. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  7. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  8. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  9. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  10. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  11. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  12. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  13. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  14. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  15. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  16. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  17. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  18. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  19. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  20. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  21. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  22. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  23. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  24. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  25. 2019DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehensioncited by 156
  26. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  27. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 98
  28. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  29. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  30. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 84
  31. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
  32. 2021Earnings-21: A Practical Benchmark for ASR in the Wildcited by 60
  33. 2023LLaSM: Large Language and Speech Modelcited by 56
  34. 2022Earnings-22: A Practical Benchmark for Accents in the Wildcited by 46
  35. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  36. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language modelscited by 35
  37. 2023Investigating the Emergent Audio Classification Ability of ASR Foundation Modelscited by 10
  38. 2023UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructionscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 58; 2026: 55.

124 papers.

Sort:

By area:

  1. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  2. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  3. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 113
  4. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  5. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  6. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  7. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  8. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 60
  9. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  10. 2026τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domainscited by 30
  11. 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
  12. 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
38 more
  1. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  2. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 22
  3. 2024SpeechPrune: Context-aware Token Pruning for Speech Information Retrievalcited by 20
  4. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMscited by 19
  5. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  6. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interactioncited by 18
  7. 2025Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Modelscited by 17
  8. 2024Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialoguecited by 16
  9. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  10. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 15
  11. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 15
  12. 2025VoiceAgentBench: Are Voice Assistants ready for agentic tasks?cited by 15
  13. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  14. 2025SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Modelscited by 14
  15. 2024Enabling Auditory Large Language Models for Automatic Speech Quality Evaluationcited by 13
  16. 2025QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptionscited by 13
  17. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 12
  18. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 11
  19. 2025Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoningcited by 10
  20. 2025EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Modelscited by 10
  21. 2025STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligencecited by 10
  22. 2025Thinking with Sound: Audio Chain-of-Thought Enables Multimodal Reasoning in Large Audio-Language Modelscited by 9
  23. 2025SpokenNativQA: Multilingual Everyday Spoken Queries for LLMscited by 8
  24. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 7
  25. 2025WearVox: An Egocentric Multichannel Voice Assistant Benchmark for Wearablescited by 7
  26. 2026Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Usecited by 7
  27. 2024Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistantcited by 6
  28. 2025Soundwave: Less is More for Speech-Text Alignment in LLMscited by 6
  29. 2025StressTest: Can YOUR Speech LM Handle the Stress?cited by 6
  30. 2025MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Taskscited by 6
  31. 2025Do Audio-Visual Segmentation Models Truly Segment Sounding Objects?cited by 5
  32. 2025Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thoughtcited by 5
  33. 2025Multi-Domain Audio Question Answering Benchmark Toward Acoustic Content Reasoningcited by 5
  34. 2025Audio Flamingo Sound-CoT Technical Report: Improving Chain-of-Thought Reasoning in Sound Understandingcited by 5
  35. 2025Robustness assessment of large audio language models in multiple-choice evaluationcited by 5
  36. 2026All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluationcited by 5
  37. 2024What Are They Doing? Joint Audio-Speech Co-Reasoningcited by 4
  38. 2025Contextual Paralinguistic Data Creation for Multi-Modal Speech-LLM: Data Condensation and Spoken QA Generationcited by 4

50 most cited of 124 All 124 (JSON)