MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models
Not on the map. Why?
Abstract (arXiv)
Large audio-language models (LALMs) have shown promising progress in understanding speech, music, and general sound events, yet their ability to reason about how audio signals are degraded remains underexplored. Existing benchmarks primarily evaluate semantic understanding, event recognition, or high-level audio reasoning, leaving a basic question unanswered: Do LALMs understand the differences in audio quality? We introduce MRMAD, a Multi-Round Multi-Audio Degradation benchmark for evaluating audio degradation perception and understanding in LALMs. MRMAD spans speech, music, and sound, and frames evaluation as multi-turn dialogues across multiple audio inputs, requiring models to identify types of degradation, compare severity, and perceive corruption changes across turns. Unlike current single-turn audio-language benchmarks, MRMAD evaluates whether LALMs can maintain consistent degradation hypotheses with new evidence and comprehend low-level acoustic phenomena over multi-turn dialogues. Through a systematic evaluation of 18 representative LALMs from non-thinking to reasoning and Omni models, we find that current models often recognize coarse content while failing to diagnose, compare, or reason about degradations reliably. Human evaluations further reveal a significant perception gap between LALMs and human listeners. MRMAD thus exposes a critical yet overlooked aspect of audio-language understanding and provides a diagnostic foundation for building future LALMs that are robust to real-world acoustic conditions.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 72 references that are in the map. First posted on arXiv 23 Aug 2026.
Builds on
34 of 72 references.
References · 34
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
- 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
- 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
- 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
- 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
- 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
- 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
- 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
- 2016FMA: A Dataset For Music Analysiscited by 207
22 more
- 2017SEGAN: Speech Enhancement Generative Adversarial Networkcited by 166
- 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 156
- 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
- 2024Large Language Models Can Learn Temporal Reasoningcited by 111
- 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 103
- 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 65
- 2016Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applicationscited by 61
- 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
- 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 41
- 2025QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Languagecited by 18
- 2025Reasoning-Enhanced Domain-Adaptive Pretraining of Multimodal Large Language Models for Short Video Content Governancecited by 16
- 2025QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptionscited by 13
- 2026Adaptive Information Control for Search-Augmented LLM Reasoningcited by 11
- 2026Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptationcited by 7
- 2026AuTAgent: A Reinforcement Learning Framework for Tool-Augmented Audio Reasoningcited by 6
- 2026A Survey of Advancing Audio Super-Resolution and Bandwidth Extension from Discriminative to Generative Modelscited by 6
- 2025Embedding-based Retrieval in Multimodal Content Moderationcited by 4
- 2026AudioRouter: Data Efficient Audio Understanding via RL based Dual Reasoningcited by 3
- 2026SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral Contextscited by 2
- 2026Why Can't They Remember? Uncovering Representation and Retrieval Bottlenecks in Multi-Turn Acoustic Memorycited by 2
- 2026Audio-Mind: An Auditable Agentic Framework for Audio Understandingcited by 2
- 2026FSD50K-Solo: Automated Curation of Single-Source Sound Eventscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026A Closer Look at Failure Modes in Temporal Understanding of Large Audio-Language Models
- 2026MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models
- 2026All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation
- 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs
- 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning
- 2025Towards Reliable Large Audio Language Model
- 2026SonicBench: Dissecting the Physical Perception Bottleneck in Large Audio Language Models
- 2025S2SBench: A Benchmark for Quantifying Intelligence Degradation in Speech-to-Speech Large Language Models