mlsanityData to 7 Oct 2026

WER we are and WER we think we are

arXiv 2010.03432 ↗First posted 7 Oct 2020cs.CL cs.LG cs.SDcited by 19

Not on the map. Why?

Abstract (arXiv)

Natural language processing of conversational speech requires the availability of high-quality transcripts. In this paper, we express our skepticism towards the recent reports of very low Word Error Rates (WERs) achieved by modern Automatic Speech Recognition (ASR) systems on benchmark datasets. We outline several problems with popular benchmarks and compare three state-of-the-art commercial ASR systems on an internal dataset of real-life spontaneous human conversations and HUB'05 public benchmark. We show that WERs are significantly higher than the best reported results. We formulate a set of guidelines which may aid in the creation of real-life, multi-domain datasets with high quality annotations for training and testing of robust ASR systems.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 1; 2023: 5; 2024: 1; 2025: 2; 2026: 5.

19 papers.

Sort:

By area:

  1. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  2. 2021Earnings-21: A Practical Benchmark for ASR in the Wildcited by 60
  3. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 41
  4. 2023The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarioscited by 20
  5. 2023Putting Natural in Natural Language Processingcited by 17
  6. 2021VarArray: Array-Geometry-Agnostic Continuous Speech Separationcited by 10
  7. 2021CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcriptioncited by 9
  8. 2023The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systemscited by 9
  9. 2022Language technology practitioners as language managers: arbitrating data bias and predictive bias in ASRcited by 7
  10. 2021Are E2E ASR models ready for an industrial usage?cited by 3
  11. 2023Using Kaldi for Automatic Speech Recognition of Conversational Austrian Germancited by 2
  12. 2025Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challengescited by 2
7 more
  1. 2025SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Modelscited by 2
  2. 2024Framework for Curating Speech Datasets and Evaluating ASR Systems: A Case Study for Polishcited by 0
  3. 2026Graph-Based Phonetic Error Correction of Noisy ASRcited by 0
  4. 2026Voice Memory for Agentic Speech Recognitioncited by 0
  5. 2026Towards Quantifying Benchmark Optimization in ASR Modelscited by 0
  6. 2026No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpuscited by 0
  7. 2026The Hidden Cost of Digits: Number Normalization and WER in ASR Systemscited by 0