# WER we are and WER we think we are

arXiv: https://arxiv.org/abs/2010.03432 · HTML: https://mlsanity.com/paper/2010.03432/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2020.

## Built on by (most cited first, 19 of 19)

- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Earnings-21: A Practical Benchmark for ASR in the Wild](https://mlsanity.com/paper/2104.11348/) (2021)
- [HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models](https://mlsanity.com/paper/2309.15701/) (2023)
- [The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios](https://mlsanity.com/paper/2306.13734/) (2023)
- [Putting Natural in Natural Language Processing](https://mlsanity.com/paper/2305.04572/) (2023)
- [VarArray: Array-Geometry-Agnostic Continuous Speech Separation](https://mlsanity.com/paper/2110.05745/) (2021)
- [CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription](https://mlsanity.com/paper/2107.01091/) (2021)
- [The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systems](https://mlsanity.com/paper/2307.15493/) (2023)
- [Language technology practitioners as language managers: arbitrating data bias and predictive bias in ASR](https://mlsanity.com/paper/2202.12603/) (2022)
- [Are E2E ASR models ready for an industrial usage?](https://mlsanity.com/paper/2112.12572/) (2021)
- [Using Kaldi for Automatic Speech Recognition of Conversational Austrian German](https://mlsanity.com/paper/2301.06475/) (2023)
- [Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challenges](https://mlsanity.com/paper/2507.18161/) (2025)
- [SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models](https://mlsanity.com/paper/2507.19361/) (2025)
- [Framework for Curating Speech Datasets and Evaluating ASR Systems: A Case Study for Polish](https://mlsanity.com/paper/2408.00005/) (2024)
- [Graph-Based Phonetic Error Correction of Noisy ASR](https://mlsanity.com/paper/2606.24889/) (2026)
- [Voice Memory for Agentic Speech Recognition](https://mlsanity.com/paper/2607.26410/) (2026)
- [Towards Quantifying Benchmark Optimization in ASR Models](https://mlsanity.com/paper/2608.19936/) (2026)
- [No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus](https://mlsanity.com/paper/2608.28875/) (2026)
- [The Hidden Cost of Digits: Number Normalization and WER in ASR Systems](https://mlsanity.com/paper/2609.21084/) (2026)

## Abstract (arXiv)

Natural language processing of conversational speech requires the availability of high-quality transcripts. In this paper, we express our skepticism towards the recent reports of very low Word Error Rates (WERs) achieved by modern Automatic Speech Recognition (ASR) systems on benchmark datasets. We outline several problems with popular benchmarks and compare three state-of-the-art commercial ASR systems on an internal dataset of real-life spontaneous human conversations and HUB'05 public benchmark. We show that WERs are significantly higher than the best reported results. We formulate a set of guidelines which may aid in the creation of real-life, multi-domain datasets with high quality annotations for training and testing of robust ASR systems.