# ASR4REAL: An extended benchmark for speech models

arXiv: https://arxiv.org/abs/2110.08583 · HTML: https://mlsanity.com/paper/2110.08583/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 16 Oct 2021.

## Builds on (most cited first, 9 of 9)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [Rethinking Evaluation in ASR: Are Our Models Robust Enough?](https://mlsanity.com/paper/2010.11745/) (2020)
- [Earnings-21: A Practical Benchmark for ASR in the Wild](https://mlsanity.com/paper/2104.11348/) (2021)
- [Toward Fairness in AI for People with Disabilities: A Research Roadmap](https://mlsanity.com/paper/1907.02227/) (2019)
- [NISP: A Multi-lingual Multi-accent Dataset for Speaker Profiling](https://mlsanity.com/paper/2007.06021/) (2020)
- [Artificial Intelligence Fairness in the Context of Accessibility Research on Intelligent Systems for People who are Deaf or Hard of Hearing](https://mlsanity.com/paper/1908.10414/) (2019)

## Built on by (most cited first, 14 of 14)

- [How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communications](https://mlsanity.com/paper/2203.16822/) (2022)
- [Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparities](https://mlsanity.com/paper/2207.11345/) (2022)
- [Towards measuring fairness in speech recognition: Fair-Speech dataset](https://mlsanity.com/paper/2408.12734/) (2024)
- [Generative Spoken Language Model based on continuous word-sized audio tokens](https://mlsanity.com/paper/2310.05224/) (2023)
- [Improving Fairness and Robustness in End-to-End Speech Recognition through unsupervised clustering](https://mlsanity.com/paper/2306.06083/) (2023)
- [Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition](https://mlsanity.com/paper/2310.18877/) (2023)
- [Summarizing Speech: A Comprehensive Survey](https://mlsanity.com/paper/2504.08024/) (2025)
- [Some voices are too common: Building fair speech recognition systems using the Common Voice dataset](https://mlsanity.com/paper/2306.03773/) (2023)
- [More Speaking or More Speakers?](https://mlsanity.com/paper/2211.00854/) (2022)
- [Multi-Variant Consistency based Self-supervised Learning for Robust Automatic Speech Recognition](https://mlsanity.com/paper/2112.12522/) (2021)
- [Pushing the performances of ASR models on English and Spanish accents](https://mlsanity.com/paper/2212.12048/) (2022)
- [StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Control](https://mlsanity.com/paper/2603.07599/) (2026)
- [Automatic Data Augmentation for Domain Adapted Fine-Tuning of Self-Supervised Speech Representations](https://mlsanity.com/paper/2306.00481/) (2023)
- [Queer inclusion in speech datasets: An audit and taxonomy of practical tensions](https://mlsanity.com/paper/2609.25491/) (2026)

## Abstract (arXiv)

Popular ASR benchmarks such as Librispeech and Switchboard are limited in the diversity of settings and speakers they represent. We introduce a set of benchmarks matching real-life conditions, aimed at spotting possible biases and weaknesses in models. We have found out that even though recent models do not seem to exhibit a gender bias, they usually show important performance discrepancies by accent, and even more important ones depending on the socio-economic status of the speakers. Finally, all tested models show a strong performance drop when tested on conversational speech, and in this precise context even a language model trained on a dataset as big as Common Crawl does not seem to have significant positive effect which reiterates the importance of developing conversational language models