# Earnings-21: A Practical Benchmark for ASR in the Wild arXiv: https://arxiv.org/abs/2104.11348 ยท HTML: https://mlsanity.com/paper/2104.11348/ ## Key facts - Cited by 62 papers in the map (data through 14 Oct 2026). - Builds on 8 of its 20 references that are in the map. - First posted on arXiv 22 Apr 2021. ## Builds on (most cited first, 8 of 8) - [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015) - [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018) - [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019) - [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018) - [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015) - [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020) - [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020) - [WER we are and WER we think we are](https://mlsanity.com/paper/2010.03432/) (2020) ## Built on by (most cited first, 30 of 62) - [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022) - [AudioBench: A Universal Benchmark for Audio Large Language Models](https://mlsanity.com/paper/2406.16020/) (2024) - [The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage](https://mlsanity.com/paper/2111.09344/) (2021) - [Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://mlsanity.com/paper/2311.00430/) (2023) - [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021) - [Earnings-22: A Practical Benchmark for Accents in the Wild](https://mlsanity.com/paper/2203.15591/) (2022) - [Voxtral](https://mlsanity.com/paper/2507.13264/) (2025) - [Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling](https://mlsanity.com/paper/2509.08753/) (2025) - [Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation](https://mlsanity.com/paper/2510.06961/) (2025) - [A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding](https://mlsanity.com/paper/2211.05869/) (2022) - [Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models](https://mlsanity.com/paper/2502.12414/) (2025) - [Improving Contextual Recognition of Rare Words with an Alternate Spelling Prediction Model](https://mlsanity.com/paper/2209.01250/) (2022) - [Semantic-WER: A Unified Metric for the Evaluation of ASR Transcript for End Usability](https://mlsanity.com/paper/2106.02016/) (2021) - [ASR4REAL: An extended benchmark for speech models](https://mlsanity.com/paper/2110.08583/) (2021) - [Improving Neural Biasing for Contextual Speech Recognition by Early Context Injection and Text Perturbation](https://mlsanity.com/paper/2407.10303/) (2024) - [VarArray: Array-Geometry-Agnostic Continuous Speech Separation](https://mlsanity.com/paper/2110.05745/) (2021) - [Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models](https://mlsanity.com/paper/2210.15734/) (2022) - [Updated Corpora and Benchmarks for Long-Form Speech Recognition](https://mlsanity.com/paper/2309.15013/) (2023) - [On the Evaluation of Speech Foundation Models for Spoken Language Understanding](https://mlsanity.com/paper/2406.10083/) (2024) - [MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Models](https://mlsanity.com/paper/2412.09818/) (2024) - [Voxtral Realtime](https://mlsanity.com/paper/2602.11298/) (2026) - [Toward Zero Oracle Word Error Rate on the Switchboard Benchmark](https://mlsanity.com/paper/2206.06192/) (2022) - [Implementing contextual biasing in GPU decoder for online ASR](https://mlsanity.com/paper/2306.15685/) (2023) - [SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluation](https://mlsanity.com/paper/2403.08196/) (2024) - [Longer is (Not Necessarily) Stronger: Punctuated Long-Sequence Training for Enhanced Speech Recognition and Translation](https://mlsanity.com/paper/2409.05601/) (2024) - [TurboBias: Universal ASR Context-Biasing powered by GPU-accelerated Phrase-Boosting Tree](https://mlsanity.com/paper/2508.07014/) (2025) - [Better Transcription of UK Supreme Court Hearings](https://mlsanity.com/paper/2211.17094/) (2022) - [Continuously Learning New Words in Automatic Speech Recognition](https://mlsanity.com/paper/2401.04482/) (2024) - [Reverb: Open-Source ASR and Diarization from Rev](https://mlsanity.com/paper/2410.03930/) (2024) - [SDBench: A Comprehensive Benchmark Suite for Speaker Diarization](https://mlsanity.com/paper/2507.16136/) (2025) ## Abstract (arXiv) Commonly used speech corpora inadequately challenge academic and commercial ASR systems. In particular, speech corpora lack metadata needed for detailed analysis and WER measurement. In response, we present Earnings-21, a 39-hour corpus of earnings calls containing entity-dense speech from nine different financial sectors. This corpus is intended to benchmark ASR systems in the wild with special attention towards named entity recognition. We benchmark four commercial ASR models, two internal models built with open-source tools, and an open-source LibriSpeech model and discuss their differences in performance on Earnings-21. Using our recently released fstalign tool, we provide a candid analysis of each model's recognition capabilities under different partitions. Our analysis finds that ASR accuracy for certain NER categories is poor, presenting a significant impediment to transcript comprehension and usage. Earnings-21 bridges academic and commercial ASR system evaluation and enables further research on entity modeling and WER on real world audio.