mlsanityData to 14 Oct 2026

Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models

arXiv 2505.19037 ↗First posted 25 May 2025eess.AS cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

We introduce Speech-IFeval, an evaluation framework designed to assess instruction-following capabilities and quantify catastrophic forgetting in speech-aware language models (SLMs). Recent SLMs integrate speech perception with large language models (LLMs), often degrading textual capabilities due to speech-centric training. Existing benchmarks conflate speech perception with instruction-following, hindering evaluation of these distinct skills. To address this gap, we provide a benchmark for diagnosing the instruction-following abilities of SLMs. Our findings show that most SLMs struggle with even basic instructions, performing far worse than text-based LLMs. Additionally, these models are highly sensitive to prompt variations, often yielding inconsistent and unreliable outputs. We highlight core challenges and provide insights to guide future research, emphasizing the need for evaluation beyond task-level metrics.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 32 of its 34 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

32 of 34 references.

References · 32

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2024Qwen2 Technical Reportcited by 2,120
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  11. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  12. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
20 more
  1. 2024Qwen2-Audio Technical Reportcited by 560
  2. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  3. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  4. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  5. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  6. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  7. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
  8. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  9. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  10. 2023Joint Audio and Speech Understandingcited by 100
  11. 2024FOFO: A Benchmark to Evaluate LLMs' Format-Following Capabilitycited by 80
  12. 2024Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Taskscited by 71
  13. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 45
  14. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 39
  15. 2024BLSP-Emo: Towards Empathetic Large Speech-Language Modelscited by 36
  16. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 33
  17. 2024Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Modelscited by 29
  18. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  19. 2024Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Modelscited by 6
  20. 2024Self-Powered LLM Modality Expansion for Large Speech-Text Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 16.

27 papers.

Sort:

By area:

  1. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  2. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 24
  3. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interactioncited by 19
  4. 2026X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMscited by 16
  5. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  6. 2026All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluationcited by 8
  7. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
  8. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  9. 2025SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Modelscited by 5
  10. 2025Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variationscited by 4
  11. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  12. 2025AHAMask: Reliable Task Specification for Large Audio Language Models without Instructionscited by 3
15 more
  1. 2025ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Modelscited by 3
  2. 2026MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Modelscited by 2
  3. 2026Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Modelscited by 2
  4. 2026How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluationcited by 2
  5. 2025TAU: A Benchmark for Cultural Sound Understanding Beyond Semanticscited by 1
  6. 2026ALARM: Audio-Language Alignment for Reasoning Modelscited by 1
  7. 2026Do What I Say: A Spoken Prompt Dataset for Instruction-Followingcited by 1
  8. 2026Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMscited by 1
  9. 2026SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Modelscited by 1
  10. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  11. 2025Understanding Textual Capability Degradation in Speech LLMs via Parameter Importance Analysiscited by 0
  12. 2026When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removalcited by 0
  13. 2026Hearing Like Humans? Sound Symbolism and Perceptual Alignment in Speech Language Modelscited by 0
  14. 2026Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Modelscited by 0
  15. 2026Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Textcited by 0