mlsanityData to 7 Oct 2026

Earnings-21: A Practical Benchmark for ASR in the Wild

arXiv 2104.11348 ↗First posted 22 Apr 2021cs.CL cs.SD eess.AScited by 60

Not on the map. Why?

Abstract (arXiv)

Commonly used speech corpora inadequately challenge academic and commercial ASR systems. In particular, speech corpora lack metadata needed for detailed analysis and WER measurement. In response, we present Earnings-21, a 39-hour corpus of earnings calls containing entity-dense speech from nine different financial sectors. This corpus is intended to benchmark ASR systems in the wild with special attention towards named entity recognition. We benchmark four commercial ASR models, two internal models built with open-source tools, and an open-source LibriSpeech model and discuss their differences in performance on Earnings-21. Using our recently released fstalign tool, we provide a candid analysis of each model's recognition capabilities under different partitions. Our analysis finds that ASR accuracy for certain NER categories is poor, presenting a significant impediment to transcript comprehension and usage. Earnings-21 bridges academic and commercial ASR system evaluation and enables further research on entity modeling and WER on real world audio.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 8 of its 20 references that are in the map. First posted on arXiv 22 Apr 2021.

Builds on

8 of 20 references.

References · 8

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  4. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  5. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  6. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  7. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  8. 2020WER we are and WER we think we arecited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 4; 2024: 14; 2025: 15; 2026: 15.

60 papers.

Sort:

By area:

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 124
  3. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 98
  4. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  5. 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
  6. 2022Earnings-22: A Practical Benchmark for Accents in the Wildcited by 46
  7. 2025Voxtralcited by 41
  8. 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 24
  9. 2025Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluationcited by 21
  10. 2022A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understandingcited by 18
  11. 2022Improving Contextual Recognition of Rare Words with an Alternate Spelling Prediction Modelcited by 17
  12. 2025Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Modelscited by 17
38 more
  1. 2021ASR4REAL: An extended benchmark for speech modelscited by 14
  2. 2021Semantic-WER: A Unified Metric for the Evaluation of ASR Transcript for End Usabilitycited by 12
  3. 2024Improving Neural Biasing for Contextual Speech Recognition by Early Context Injection and Text Perturbationcited by 11
  4. 2021VarArray: Array-Geometry-Agnostic Continuous Speech Separationcited by 10
  5. 2022Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Modelscited by 10
  6. 2023Updated Corpora and Benchmarks for Long-Form Speech Recognitioncited by 10
  7. 2024On the Evaluation of Speech Foundation Models for Spoken Language Understandingcited by 9
  8. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 7
  9. 2026Voxtral Realtimecited by 5
  10. 2022Toward Zero Oracle Word Error Rate on the Switchboard Benchmarkcited by 4
  11. 2023Implementing contextual biasing in GPU decoder for online ASRcited by 4
  12. 2024SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluationcited by 4
  13. 2024Longer is (Not Necessarily) Stronger: Punctuated Long-Sequence Training for Enhanced Speech Recognition and Translationcited by 4
  14. 2025TurboBias: Universal ASR Context-Biasing powered by GPU-accelerated Phrase-Boosting Treecited by 4
  15. 2022Better Transcription of UK Supreme Court Hearingscited by 3
  16. 2024Continuously Learning New Words in Automatic Speech Recognitioncited by 3
  17. 2024Reverb: Open-Source ASR and Diarization from Revcited by 3
  18. 2025SDBench: A Comprehensive Benchmark Suite for Speaker Diarizationcited by 3
  19. 2024Evaluating Named Entity Recognition: A comparative analysis of mono- and multilingual transformer models on a novel Brazilian corporate earnings call transcripts datasetcited by 2
  20. 2024Beyond Levenshtein: Leveraging Multiple Algorithms for Robust Word Error Rate Computations And Granular Error Classificationscited by 2
  21. 2024Quantification of stylistic differences in human- and ASR-produced transcripts of African American Englishcited by 2
  22. 2025FinAudio: A Benchmark for Audio Large Language Models in Financial Applicationscited by 2
  23. 2025Spoken Language Understanding on Unseen Tasks With In-Context Learningcited by 2
  24. 2025PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systemscited by 2
  25. 2025ConsumerBench: Benchmarking Generative AI Applications on End-User Devicescited by 2
  26. 2024Post-decoder Biasing for End-to-End Speech Recognition of Multi-turn Medical Interviewcited by 1
  27. 2024Unifying Global and Near-Context Biasing in a Single Trie Passcited by 1
  28. 2025FlexCTC: GPU-powered CTC Beam Decoding With Advanced Contextual Abilitiescited by 1
  29. 2025Retrieval Augmented Generation based context discovery for ASRcited by 1
  30. 2026ReHear: Iterative Pseudo-Label Refinement for Semi-Supervised Speech Recognition via Audio Large Language Modelscited by 1
  31. 2026Streaming Translation and Transcription Through Speech-to-Text Causal Alignmentcited by 1
  32. 2026When Misinformation Speaks and Converses: Rethinking Fact-Checking in Audio Platformscited by 1
  33. 2026How to Recognize New Words: A Comparison Between Context Biasing Methods and Speech LLMscited by 1
  34. 2023Mi-Go: Test Framework which uses YouTube as Data Source for Evaluating Speech Recognition Models like OpenAI's Whispercited by 0
  35. 2024Less is More: Accurate Speech Recognition & Translation without Web-Scale Datacited by 0
  36. 2025Context Biasing for Pronunciation-Orthography Mismatch in Automatic Speech Recognitioncited by 0
  37. 2025Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASRcited by 0
  38. 2025Segmental Attention Decoding With Long Form Acoustic Encodingscited by 0

50 most cited of 60 All 60 (JSON)