# Measuring the Accuracy of Automatic Speech Recognition Solutions

arXiv: https://arxiv.org/abs/2408.16287 · HTML: https://mlsanity.com/paper/2408.16287/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 63 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Builds on (most cited first, 18 of 18)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [MOSNet: Deep Learning based Objective Assessment for Voice Conversion](https://mlsanity.com/paper/1904.08352/) (2019)
- [Self-training and Pre-training are Complementary for Speech Recognition](https://mlsanity.com/paper/2010.11430/) (2020)
- [SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network](https://mlsanity.com/paper/2104.02133/) (2021)
- [The Microsoft 2017 Conversational Speech Recognition System](https://mlsanity.com/paper/1708.06073/) (2017)
- [English Conversational Telephone Speech Recognition by Humans and Machines](https://mlsanity.com/paper/1703.02136/) (2017)
- [Improving noise robust automatic speech recognition with single-channel time-domain enhancement network](https://mlsanity.com/paper/2003.03998/) (2020)
- [Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing](https://mlsanity.com/paper/1712.02033/) (2017)
- [Analyzing the Quality and Stability of a Streaming End-to-End On-Device Speech Recognizer](https://mlsanity.com/paper/2006.01416/) (2020)
- [You don't understand me!: Comparing ASR results for L1 and L2 speakers of Swedish](https://mlsanity.com/paper/2405.13379/) (2024)
- [Social, Environmental, and Technical: Factors at Play in the Current Use and Future Design of Small-Group Captioning](https://mlsanity.com/paper/2109.10412/) (2021)

## Built on by (most cited first, 15 of 15)

- [Evaluating ASR Confidence Scores for Automated Error Detection in User-Assisted Correction Interfaces](https://mlsanity.com/paper/2503.15124/) (2025)
- [Listening with Attention: Entropy-Guided Explainability for Transformer-Based Audio Models](https://mlsanity.com/paper/2606.14647/) (2026)
- [Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasia](https://mlsanity.com/paper/2506.08846/) (2025)
- [On the Robust Approximation of ASR Metrics](https://mlsanity.com/paper/2502.12408/) (2025)
- [Enhancing Adaptive Behavioral Interventions with LLM Inference from Participant-Described States](https://mlsanity.com/paper/2507.03871/) (2025)
- [MAGIC-Enhanced Keyword Prompting for Zero-Shot Audio Captioning with CLIP Models](https://mlsanity.com/paper/2509.12591/) (2025)
- [A Unified and Reproducible Experimentation Framework for Speech Understanding](https://mlsanity.com/paper/2605.30899/) (2026)
- [Automating Feedback Analysis in Surgical Training: Detection, Categorization, and Assessment](https://mlsanity.com/paper/2412.00760/) (2024)
- [Real-Time Sign Language Gestures to Speech Transcription using Deep Learning](https://mlsanity.com/paper/2508.12713/) (2025)
- [Mic Drop or Data Flop? Evaluating the Fitness for Purpose of AI Voice Interviewers for Data Collection within Quantitative & Qualitative Research Contexts](https://mlsanity.com/paper/2509.01814/) (2025)
- [Multilingual Lexical Feature Analysis of Spoken Language for Predicting Major Depression Symptom Severity](https://mlsanity.com/paper/2511.07011/) (2025)
- [From Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programs](https://mlsanity.com/paper/2512.19161/) (2025)
- [Unheard in the Digital Age: Rethinking AI Bias and Speech Diversity](https://mlsanity.com/paper/2601.18641/) (2026)
- [Convex Low-resource Accent-Robust Language Detection in Speech Recognition](https://mlsanity.com/paper/2605.23235/) (2026)
- [Pretrained ASR Pseudo-labeling for Noisy Police Audio](https://mlsanity.com/paper/2609.30469/) (2026)

## Abstract (arXiv)

For d/Deaf and hard of hearing (DHH) people, captioning is an essential accessibility tool. Significant developments in artificial intelligence (AI) mean that Automatic Speech Recognition (ASR) is now a part of many popular applications. This makes creating captions easy and broadly available - but transcription needs high levels of accuracy to be accessible. Scientific publications and industry report very low error rates, claiming AI has reached human parity or even outperforms manual transcription. At the same time the DHH community reports serious issues with the accuracy and reliability of ASR. There seems to be a mismatch between technical innovations and the real-life experience for people who depend on transcription. Independent and comprehensive data is needed to capture the state of ASR. We measured the performance of eleven common ASR services with recordings of Higher Education lectures. We evaluated the influence of technical conditions like streaming, the use of vocabularies, and differences between languages. Our results show that accuracy ranges widely between vendors and for the individual audio samples. We also measured a significant lower quality for streaming ASR, which is used for live events. Our study shows that despite the recent improvements of ASR, common services lack reliability in accuracy.