mlsanityData to 14 Oct 2026

Measuring the Accuracy of Automatic Speech Recognition Solutions

arXiv 2408.16287 ↗First posted 29 Aug 2024cs.CL cs.SD eess.AScited by 15

Not on the map. Why?

Abstract (arXiv)

For d/Deaf and hard of hearing (DHH) people, captioning is an essential accessibility tool. Significant developments in artificial intelligence (AI) mean that Automatic Speech Recognition (ASR) is now a part of many popular applications. This makes creating captions easy and broadly available - but transcription needs high levels of accuracy to be accessible. Scientific publications and industry report very low error rates, claiming AI has reached human parity or even outperforms manual transcription. At the same time the DHH community reports serious issues with the accuracy and reliability of ASR. There seems to be a mismatch between technical innovations and the real-life experience for people who depend on transcription. Independent and comprehensive data is needed to capture the state of ASR. We measured the performance of eleven common ASR services with recordings of Higher Education lectures. We evaluated the influence of technical conditions like streaming, the use of vocabularies, and differences between languages. Our results show that accuracy ranges widely between vendors and for the individual audio samples. We also measured a significant lower quality for streaming ASR, which is used for live events. Our study shows that despite the recent improvements of ASR, common services lack reliability in accuracy.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 18 of its 63 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

18 of 63 references.

References · 18

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  4. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  5. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  6. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  7. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  8. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  9. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 127
  10. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 99
  11. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  12. 2017The Microsoft 2017 Conversational Speech Recognition Systemcited by 52
6 more
  1. 2017English Conversational Telephone Speech Recognition by Humans and Machinescited by 40
  2. 2020Improving noise robust automatic speech recognition with single-channel time-domain enhancement networkcited by 21
  3. 2017Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearingcited by 11
  4. 2020Analyzing the Quality and Stability of a Streaming End-to-End On-Device Speech Recognizercited by 7
  5. 2024You don't understand me!: Comparing ASR results for L1 and L2 speakers of Swedishcited by 5
  6. 2021Social, Environmental, and Technical: Factors at Play in the Current Use and Future Design of Small-Group Captioningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 5.

15 papers.

Sort:

  1. 2025Evaluating ASR Confidence Scores for Automated Error Detection in User-Assisted Correction Interfacescited by 6
  2. 2026Listening with Attention: Entropy-Guided Explainability for Transformer-Based Audio Modelscited by 4
  3. 2025Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasiacited by 3
  4. 2025On the Robust Approximation of ASR Metricscited by 2
  5. 2025Enhancing Adaptive Behavioral Interventions with LLM Inference from Participant-Described Statescited by 1
  6. 2025MAGIC-Enhanced Keyword Prompting for Zero-Shot Audio Captioning with CLIP Modelscited by 1
  7. 2026A Unified and Reproducible Experimentation Framework for Speech Understandingcited by 1
  8. 2024Automating Feedback Analysis in Surgical Training: Detection, Categorization, and Assessmentcited by 0
  9. 2025Real-Time Sign Language Gestures to Speech Transcription using Deep Learningcited by 0
  10. 2025Mic Drop or Data Flop? Evaluating the Fitness for Purpose of AI Voice Interviewers for Data Collection within Quantitative & Qualitative Research Contextscited by 0
  11. 2025Multilingual Lexical Feature Analysis of Spoken Language for Predicting Major Depression Symptom Severitycited by 0
  12. 2025From Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programscited by 0
3 more
  1. 2026Unheard in the Digital Age: Rethinking AI Bias and Speech Diversitycited by 0
  2. 2026Convex Low-resource Accent-Robust Language Detection in Speech Recognitioncited by 0
  3. 2026Pretrained ASR Pseudo-labeling for Noisy Police Audiocited by 0