mlsanityData to 7 Oct 2026

Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech

arXiv 2309.09510 ↗First posted 18 Sep 2023eess.AS cs.LG cs.SDcited by 99

Not on the map. Why?

Abstract (arXiv)

Text language models have shown remarkable zero-shot capability in generalizing to unseen tasks when provided with well-formulated instructions. However, existing studies in speech processing primarily focus on limited or specific tasks. Moreover, the lack of standardized benchmarks hinders a fair comparison across different approaches. Thus, we present Dynamic-SUPERB, a benchmark designed for building universal speech models capable of leveraging instruction tuning to perform multiple tasks in a zero-shot fashion. To achieve comprehensive coverage of diverse speech tasks and harness instruction tuning, we invite the community to collaborate and contribute, facilitating the dynamic growth of the benchmark. To initiate, Dynamic-SUPERB features 55 evaluation instances by combining 33 tasks and 22 datasets. This spans a broad spectrum of dimensions, providing a comprehensive platform for evaluation. Additionally, we propose several approaches to establish benchmark baselines. These include the utilization of speech models, text language models, and the multimodal encoder. Evaluation results indicate that while these baselines perform reasonably on seen tasks, they struggle with unseen ones. We release all materials to the public and welcome researchers to collaborate on the project, advancing technologies in the field together.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Builds on 36 of its 43 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

36 of 43 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
24 more
  1. 2021GPT Understands, Toocited by 715
  2. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  3. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  4. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  5. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  7. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  8. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  9. 2023Listen, Think, and Understandcited by 206
  10. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  11. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  12. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  13. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  14. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 70
  15. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
  16. 2021LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speechcited by 46
  17. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  18. 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
  19. 2023SpeechPrompt v2: Prompt Tuning for Speech Classification Taskscited by 31
  20. 2022AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Taskscited by 29
  21. 2022Exploring Efficient-tuning Methods in Self-supervised Speech Modelscited by 26
  22. 2023Can ChatGPT Detect Intent? Evaluating Large Language Models for Spoken Language Understandingcited by 21
  23. 2023SpeechGen: Unlocking the Generative Power of Speech Language Models with Promptscited by 21
  24. 2022ESB: A Benchmark For Multi-Domain End-to-End Speech Recognitioncited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 26; 2025: 38; 2026: 32.

99 papers.

Sort:

By area:

  1. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  2. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  3. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  4. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 124
  5. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  6. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 113
  7. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  8. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
  9. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  10. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 60
  11. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  12. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
38 more
  1. 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 45
  2. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 43
  3. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  4. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  5. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 32
  6. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 30
  7. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  8. 2024Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Modelscited by 28
  9. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  10. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  11. 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
  12. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  13. 2024Building a Taiwanese Mandarin Spoken Language Model: A First Attemptcited by 22
  14. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 22
  15. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 18
  16. 2025S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Modelscited by 16
  17. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 15
  18. 2025SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuningcited by 14
  19. 2023Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervisioncited by 13
  20. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  21. 2024Enabling Auditory Large Language Models for Automatic Speech Quality Evaluationcited by 13
  22. 2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
  23. 2025BLAB: Brutally Long Audio Benchcited by 11
  24. 2025Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samplescited by 11
  25. 2025Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Modelscited by 11
  26. 2025EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Modelscited by 10
  27. 2023UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructionscited by 9
  28. 2024What Do Speech Foundation Models Not Learn About Speech?cited by 8
  29. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
  30. 2024SpeechCaps: Advancing Instruction-Based Universal Speech Models with Multi-Talker Speaking Style Captioningcited by 6
  31. 2024Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Modelscited by 6
  32. 2025Soundwave: Less is More for Speech-Text Alignment in LLMscited by 6
  33. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  34. 2025Robustness assessment of large audio language models in multiple-choice evaluationcited by 5
  35. 2026The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASRrightarrowLLM Pipelines?cited by 5
  36. 2026All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluationcited by 5
  37. 2024STAB: Speech Tokenizer Assessment Benchmarkcited by 4
  38. 2024What Are They Doing? Joint Audio-Speech Co-Reasoningcited by 4

50 most cited of 99 All 99 (JSON)