mlsanityData to 7 Oct 2026

AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension

arXiv 2402.07729 ↗First posted 12 Feb 2024eess.AS cs.CL cs.LGcited by 215

Not on the map. Why?

Cited by 32 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, instruction-following audio-language models have received broad attention for human-audio interaction. However, the absence of benchmarks capable of evaluating audio-centric interaction capabilities has impeded advancements in this field. Previous models primarily focus on assessing different fundamental tasks, such as Automatic Speech Recognition (ASR), and lack an assessment of the open-ended generative capabilities centered around audio. Thus, it is challenging to track the progression in the Large Audio-Language Models (LALMs) domain and to provide guidance for future improvement. In this paper, we introduce AIR-Bench (Audio InstRuction Benchmark), the first benchmark designed to evaluate the ability of LALMs to understand various types of audio signals (including human speech, natural sounds, and music), and furthermore, to interact with humans in the textual format. AIR-Bench encompasses two dimensions: textitfoundation and textitchat benchmarks. The former consists of 19 tasks with approximately 19k single-choice questions, intending to inspect the basic single-task ability of LALMs. The latter one contains 2k instances of open-ended question-and-answer data, directly assessing the comprehension of the model on complex audio and its capacity to follow instructions. Both benchmarks require the model to generate hypotheses directly. We design a unified framework that leverages advanced language models, such as GPT-4, to evaluate the scores of generated hypotheses given the meta-information of the audio. Experimental results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation. By revealing the limitations of existing LALMs through evaluation results, AIR-Bench can provide insights into the direction of future research.

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). Builds on 30 of its 62 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

30 of 62 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  8. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  9. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  11. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  12. 2019Clotho: An Audio Captioning Datasetcited by 354
18 more
  1. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  2. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  3. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  4. 2016FMA: A Dataset For Music Analysiscited by 205
  5. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  6. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  7. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  8. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  9. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  10. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 84
  11. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  12. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 75
  13. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  14. 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
  15. 2023LLaSM: Large Language and Speech Modelcited by 56
  16. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 54
  17. 2022CochlScene: Acquisition of acoustic scene data using crowdsourcingcited by 33
  18. 2021Text-to-Audio Grounding: Building Correspondence Between Captions and Sound Eventscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 103; 2026: 84.

215 papers.

Sort:

By area:

  1. 2024Qwen2-Audio Technical Reportcited by 543
  2. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  3. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  4. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  5. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  6. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 124
  7. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  8. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 113
  9. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  10. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  11. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  12. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
38 more
  1. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
  2. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  3. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  4. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  5. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  6. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  7. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 60
  8. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  9. 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 45
  10. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 43
  11. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 43
  12. 2024Baichuan-Omni Technical Reportcited by 42
  13. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  14. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  15. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  16. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
  17. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 30
  18. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  19. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 29
  20. 2024Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Modelscited by 28
  21. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  22. 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
  23. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  24. 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
  25. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  26. 2024Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarkingcited by 23
  27. 2024Lyra: An Efficient and Speech-Centric Framework for Omni-Cognitioncited by 20
  28. 2025AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluationcited by 20
  29. 2025Audio-Thinker: Guiding Audio Language Model When and How to Think via Reinforcement Learningcited by 19
  30. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMscited by 19
  31. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  32. 2025Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Modelscited by 17
  33. 2025VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Modelscited by 16
  34. 2025S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Modelscited by 16
  35. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  36. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 15
  37. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 15
  38. 2025VoiceAgentBench: Are Voice Assistants ready for agentic tasks?cited by 15

50 most cited of 215 All 215 (JSON)