# AudioBench: A Universal Benchmark for Audio Large Language Models

arXiv: https://arxiv.org/abs/2406.16020 · HTML: https://mlsanity.com/paper/2406.16020/

## Key facts

- Cited by 128 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 78 references that are in the map.
- First posted on arXiv 23 Jun 2024.

## Builds on (most cited first, 30 of 50)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)

## Built on by (most cited first, 30 of 128)

- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix](https://mlsanity.com/paper/2505.13032/) (2025)
- [MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2506.04779/) (2025)
- [WavChat: A Survey of Spoken Dialogue Models](https://mlsanity.com/paper/2411.13577/) (2024)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [Recent Advances in Speech Language Models: A Survey](https://mlsanity.com/paper/2410.03751/) (2024)
- [Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model](https://mlsanity.com/paper/2501.07246/) (2025)
- [DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment](https://mlsanity.com/paper/2507.02768/) (2025)
- [τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains](https://mlsanity.com/paper/2603.13686/) (2026)
- [Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models](https://mlsanity.com/paper/2505.19037/) (2025)
- [Audio Large Language Models Can Be Descriptive Speech Quality Evaluators](https://mlsanity.com/paper/2501.17202/) (2025)
- [Salmon: A Suite for Acoustic Language Model Evaluation](https://mlsanity.com/paper/2409.07437/) (2024)
- [Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decoding](https://mlsanity.com/paper/2506.07233/) (2025)
- [Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Models](https://mlsanity.com/paper/2501.13772/) (2025)
- [AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs](https://mlsanity.com/paper/2510.07293/) (2025)
- [SpeechPrune: Context-aware Token Pruning for Speech Information Retrieval](https://mlsanity.com/paper/2412.12009/) (2024)
- [VoiceAgentBench: Are Voice Assistants ready for agentic tasks?](https://mlsanity.com/paper/2510.07978/) (2025)
- [Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction](https://mlsanity.com/paper/2512.14865/) (2025)
- [AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Models](https://mlsanity.com/paper/2506.05140/) (2025)
- [VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Models](https://mlsanity.com/paper/2505.15727/) (2025)
- [CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following](https://mlsanity.com/paper/2506.12285/) (2025)
- [Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue](https://mlsanity.com/paper/2409.04927/) (2024)
- [WavReward: Spoken Dialogue Models With Generalist Reward Evaluators](https://mlsanity.com/paper/2505.09558/) (2025)
- [MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks](https://mlsanity.com/paper/2507.19634/) (2025)
- [SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models](https://mlsanity.com/paper/2508.02018/) (2025)
- [Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation](https://mlsanity.com/paper/2409.16644/) (2024)
- [QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions](https://mlsanity.com/paper/2503.20290/) (2025)
- [SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models](https://mlsanity.com/paper/2603.16859/) (2026)
- [MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models](https://mlsanity.com/paper/2604.12928/) (2026)

## Abstract (arXiv)

We introduce AudioBench, a universal benchmark designed to evaluate Audio Large Language Models (AudioLLMs). It encompasses 8 distinct tasks and 26 datasets, among which, 7 are newly proposed datasets. The evaluation targets three main aspects: speech understanding, audio scene understanding, and voice understanding (paralinguistic). Despite recent advancements, there lacks a comprehensive benchmark for AudioLLMs on instruction following capabilities conditioned on audio signals. AudioBench addresses this gap by setting up datasets as well as desired evaluation metrics. Besides, we also evaluated the capabilities of five popular models and found that no single model excels consistently across all tasks. We outline the research outlook for AudioLLMs and anticipate that our open-sourced evaluation toolkit, data, and leaderboard will offer a robust testbed for future model developments.