mlsanityData to 14 Oct 2026

Speech Robust Bench: A Robustness Benchmark For Speech Recognition

arXiv 2403.07937 ↗First posted 8 Mar 2024eess.AS cs.CL cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

As Automatic Speech Recognition (ASR) models become ever more pervasive, it is important to ensure that they make reliable predictions under corruptions present in the physical and digital world. We propose Speech Robust Bench (SRB), a comprehensive benchmark for evaluating the robustness of ASR models to diverse corruptions. SRB is composed of 114 input perturbations which simulate an heterogeneous range of corruptions that ASR models may encounter when deployed in the wild. We use SRB to evaluate the robustness of several state-of-the-art ASR models and observe that model size and certain modeling choices such as the use of discrete representations, or self-training appear to be conducive to robustness. We extend this analysis to measure the robustness of ASR models on data from various demographic subgroups, namely English and Spanish speakers, and males and females. Our results revealed noticeable disparities in the model's robustness across subgroups. We believe that SRB will significantly facilitate future research towards robust ASR models, by making it easier to conduct comprehensive and comparable robustness evaluations.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 53 of its 87 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

53 of 87 references.

References · 53

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  3. 2013Intriguing properties of neural networkscited by 5,155
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  9. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  10. 2019Natural Adversarial Examplescited by 1,408
  11. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  12. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
38 more
  1. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 685
  2. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
  3. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  4. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  5. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  6. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  7. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  8. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  9. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 446
  10. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  11. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  12. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  13. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 232
  14. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  15. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 212
  16. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  17. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 156
  18. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 155
  19. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  20. 2020Perceptual Adversarial Robustness: Defense Against Unseen Threat Modelscited by 137
  21. 2020The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Resultscited by 133
  22. 2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
  23. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  24. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 99
  25. 2018The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselinescited by 92
  26. 2020Continuous speech separation: dataset and analysiscited by 74
  27. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 73
  28. 2023SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agentscited by 71
  29. 2020Fairness Through Robustness: Investigating Robustness Disparity in Deep Learningcited by 56
  30. 2017Feature-Guided Black-Box Safety Testing of Deep Neural Networkscited by 49
  31. 2019A scalable noisy speech dataset and online subjective test frameworkcited by 48
  32. 2019Universal Adversarial Perturbations for Speech Recognition Systemscited by 41
  33. 2022Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Datacited by 30
  34. 2021Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptionscited by 26
  35. 2021The Accented English Speech Recognition Challenge 2020: Open Datasets, Tracks, Baselines, Results and Methodscited by 24
  36. 2020Towards Query-Efficient Black-Box Adversary with Zeroth-Order Natural Gradient Descentcited by 18
  37. 2022Query Efficient Decision Based Sparse Attacks Against Black-Box Deep Learning Modelscited by 13
  38. 2022Recent improvements of ASR models in the face of adversarial attackscited by 13

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 12.

21 papers.

Sort:

By area:

  1. 2025ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpuscited by 10
  2. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 7
  3. 2025SVeritas: Benchmark for Robust Speaker Verification under Diverse Conditionscited by 6
  4. 2026Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?cited by 6
  5. 2026Mega-ASR: Towards In-the-wild^2 Speech Recognition via Scaling up Real-world Acoustic Simulationcited by 5
  6. 2025The Esethu Framework: Reimagining Sustainable Dataset Governance and Curation for Low-Resource Languagescited by 4
  7. 2026Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognitioncited by 4
  8. 2025textttAVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Timecited by 2
  9. 2026RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systemscited by 2
  10. 2025ASR Under Noise: Exploring Robustness for Sundanese and Javanesecited by 1
  11. 2026SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noisecited by 1
  12. 2024Revisiting Acoustic Features for Robust ASRcited by 0
9 more
  1. 2024Incorporating Talker Identity Aids With Improving Speech Recognition in Adversarial Environmentscited by 0
  2. 2025Evaluating and Improving the Robustness of Speech Command Recognition Models to Noise and Distribution Shiftscited by 0
  3. 2026Back to Basics: Revisiting ASR in the Age of Voice Agentscited by 0
  4. 2026Benchmarking AI for low-resource contexts: Thinking beyond leaderboardscited by 0
  5. 2026ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Drivingcited by 0
  6. 2026Towards Quantifying Benchmark Optimization in ASR Modelscited by 0
  7. 2026Quieter Than the Room: Representation Drift and Task Robustness in Speech Encoderscited by 0
  8. 2026Psychoacoustically Aligned Latent Smoothing for Adversarial Robustness of Full-Duplex Speech-to-Speech Dialogue Modelscited by 0
  9. 2026Benchmarking Automatic Speech Recognition Tools for Iberian Languagescited by 0