mlsanityData to 7 Oct 2026

ASR4REAL: An extended benchmark for speech models

arXiv 2110.08583 ↗First posted 16 Oct 2021eess.AS cs.AI cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Popular ASR benchmarks such as Librispeech and Switchboard are limited in the diversity of settings and speakers they represent. We introduce a set of benchmarks matching real-life conditions, aimed at spotting possible biases and weaknesses in models. We have found out that even though recent models do not seem to exhibit a gender bias, they usually show important performance discrepancies by accent, and even more important ones depending on the socio-economic status of the speakers. Finally, all tested models show a strong performance drop when tested on conversational speech, and in this precise context even a language model trained on a dataset as big as Common Crawl does not seem to have significant positive effect which reiterates the importance of developing conversational language models

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 9 of its 22 references that are in the map. First posted on arXiv 16 Oct 2021.

Builds on

9 of 22 references.

References · 9

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  3. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  4. 2018The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselinescited by 90
  5. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 72
  6. 2021Earnings-21: A Practical Benchmark for ASR in the Wildcited by 60
  7. 2019Toward Fairness in AI for People with Disabilities: A Research Roadmapcited by 21
  8. 2020NISP: A Multi-lingual Multi-accent Dataset for Speaker Profilingcited by 7
  9. 2019Artificial Intelligence Fairness in the Context of Accessibility Research on Intelligent Systems for People who are Deaf or Hard of Hearingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 5; 2024: 1; 2025: 1; 2026: 2.

14 papers.

Sort:

  1. 2022How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communicationscited by 30
  2. 2022Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparitiescited by 27
  3. 2024Towards measuring fairness in speech recognition: Fair-Speech datasetcited by 26
  4. 2023Generative Spoken Language Model based on continuous word-sized audio tokenscited by 19
  5. 2023Improving Fairness and Robustness in End-to-End Speech Recognition through unsupervised clusteringcited by 11
  6. 2023Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognitioncited by 9
  7. 2025Summarizing Speech: A Comprehensive Surveycited by 5
  8. 2022More Speaking or More Speakers?cited by 2
  9. 2023Some voices are too common: Building fair speech recognition systems using the Common Voice datasetcited by 2
  10. 2021Multi-Variant Consistency based Self-supervised Learning for Robust Automatic Speech Recognitioncited by 1
  11. 2022Pushing the performances of ASR models on English and Spanish accentscited by 1
  12. 2026StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Controlcited by 1
2 more
  1. 2023Automatic Data Augmentation for Domain Adapted Fine-Tuning of Self-Supervised Speech Representationscited by 0
  2. 2026Queer inclusion in speech datasets: An audit and taxonomy of practical tensionscited by 0