mlsanityData to 14 Oct 2026

MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark

arXiv 2410.19168 ↗First posted 24 Oct 2024eess.AS cs.AI cs.CLcited by 288

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

The ability to comprehend audio--which includes speech, non-speech sounds, and music--is crucial for AI agents to interact effectively with the world. We present MMAU, a novel benchmark designed to evaluate multimodal audio understanding models on tasks requiring expert-level knowledge and complex reasoning. MMAU comprises 10k carefully curated audio clips paired with human-annotated natural language questions and answers spanning speech, environmental sounds, and music. It includes information extraction and reasoning questions, requiring models to demonstrate 27 distinct skills across unique and challenging tasks. Unlike existing benchmarks, MMAU emphasizes advanced perception and reasoning with domain-specific knowledge, challenging models to tackle tasks akin to those faced by experts. We assess 18 open-source and proprietary (Large) Audio-Language Models, demonstrating the significant challenges posed by MMAU. Notably, even the most advanced Gemini Pro v1.5 achieves only 52.97% accuracy, and the state-of-the-art open-source Qwen2-Audio achieves only 52.50%, highlighting considerable room for improvement. We believe MMAU will drive the audio and multimodal research community to develop more advanced audio understanding models capable of solving complex audio tasks.

Key facts. Cited by 288 papers in the map (data through 14 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 62 of its 81 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

62 of 81 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities17 Jun
  2. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models12 Oct

Other references · 60

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
38 more
  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  4. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  5. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  6. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  7. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  8. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  9. 2024Qwen2-Audio Technical Reportcited by 560
  10. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  12. 2023MusicLM: Generating Music From Textcited by 448
  13. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  14. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  15. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  16. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  17. 2023Pengi: An Audio Language Model for Audio Taskscited by 220
  18. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  19. 2023Listen, Think, and Understandcited by 209
  20. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
  21. 2023OpenAGI: When LLM Meets Domain Expertscited by 197
  22. 2024Stable Audio Opencited by 197
  23. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  24. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 184
  25. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 181
  26. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 174
  27. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
  28. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  29. 2023Joint Audio and Speech Understandingcited by 100
  30. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  31. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 96
  32. 2021The Benefit Of Temporally-Strong Labels In Audio Event Classificationcited by 83
  33. 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
  34. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  35. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 76
  36. 2023DeAR: Debiasing Vision-Language Models with Additive Residualscited by 70
  37. 2023The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluationcited by 68
  38. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 118; 2026: 166.

288 papers.

Sort:

By area:

  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  2. 2025Kimi-Audio Technical Reportcited by 222
  3. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  4. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  5. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  6. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 117
  7. 2025Step-Audio 2 Technical Reportcited by 105
  8. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 95
  9. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  10. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  11. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  12. 2025Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answeringcited by 65
38 more
  1. 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 65
  2. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 62
  3. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 59
  4. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  5. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  6. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  7. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 39
  8. 2025Step-Audio-R1 Technical Reportcited by 36
  9. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 36
  10. 2025SARI: Structured Audio Reasoning via Curriculum-Guided Reinforcement Learningcited by 34
  11. 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perceptioncited by 30
  12. 2024Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Modelscited by 29
  13. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  14. 2025Mellow: a small audio language model for reasoningcited by 27
  15. 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
  16. 2025LongCat-Flash-Omni Technical Reportcited by 26
  17. 2025FaceXBench: Evaluating Multimodal LLMs on Face Understandingcited by 24
  18. 2025EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judgecited by 24
  19. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 24
  20. 2025Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliancecited by 24
  21. 2025Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Modelscited by 23
  22. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMscited by 22
  23. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 22
  24. 2024SpeechPrune: Context-aware Token Pruning for Speech Information Retrievalcited by 21
  25. 2025AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluationcited by 21
  26. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  27. 2026Large Language Model Reasoning Failurescited by 21
  28. 2025Audio-Thinker: Guiding Audio Language Model When and How to Think via Reinforcement Learningcited by 19
  29. 2025SHANKS: Simultaneous Hearing and Thinking for Spoken Language Modelscited by 19
  30. 2025VoiceAgentBench: Are Voice Assistants ready for agentic tasks?cited by 19
  31. 2025Fun-Audio-Chat Technical Reportcited by 19
  32. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 19
  33. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  34. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 17
  35. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluationcited by 17
  36. 2025VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Modelscited by 16
  37. 2025S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Modelscited by 16
  38. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15

50 most cited of 288 All 288 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CMDAR: A Chinese Multi-scene Dynamic Audio Reasoning Benchmark with Diverse Challenges
  2. 2025Thinking with Sound: Audio Chain-of-Thought Enables Multimodal Reasoning in Large Audio-Language Models
  3. 2026Audio-Cogito: Towards Deep Audio Reasoning in Large Audio Language Models
  4. 2026AudioRAG: A Challenging Benchmark for Audio Reasoning and Information Retrieval
  5. 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
  6. 2026Multi-Source Evidence Fusion for Audio Question Answering
  7. 2025SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Models
  8. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head