# Robustness of end-to-end Automatic Speech Recognition Models -- A Case Study using Mozilla DeepSpeech

arXiv: https://arxiv.org/abs/2105.09742 · HTML: https://mlsanity.com/paper/2105.09742/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 13 references that are in the map.
- First posted on arXiv 8 May 2021.

## Builds on (most cited first, 6 of 6)

- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Rethinking Evaluation in ASR: Are Our Models Robust Enough?](https://mlsanity.com/paper/2010.11745/) (2020)
- [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019)
- [Self-Attentional Acoustic Models](https://mlsanity.com/paper/1803.09519/) (2018)
- [Massively Multilingual Adversarial Speech Recognition](https://mlsanity.com/paper/1904.02210/) (2019)
- [Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.02720/) (2017)

## Abstract (arXiv)

When evaluating the performance of automatic speech recognition models, usually word error rate within a certain dataset is used. Special care must be taken in understanding the dataset in order to report realistic performance numbers. We argue that many performance numbers reported probably underestimate the expected error rate. We conduct experiments controlling for selection bias, gender as well as overlap (between training and test data) in content, voices, and recording conditions. We find that content overlap has the biggest impact, but other factors like gender also play a role.