# Evaluating ASR Confidence Scores for Automated Error Detection in User-Assisted Correction Interfaces

arXiv: https://arxiv.org/abs/2503.15124 · HTML: https://mlsanity.com/paper/2503.15124/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 69 references that are in the map.
- First posted on arXiv 19 Mar 2025.

## Builds on (most cited first, 19 of 19)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem](https://mlsanity.com/paper/1812.05720/) (2018)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Semi-Supervised Learning with Ladder Networks](https://mlsanity.com/paper/1507.02672/) (2015)
- [Learning Confidence for Out-of-Distribution Detection in Neural Networks](https://mlsanity.com/paper/1802.04865/) (2018)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding](https://mlsanity.com/paper/2207.02971/) (2022)
- [Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition](https://mlsanity.com/paper/2010.11428/) (2020)
- [Active Learning for Speech Recognition: the Power of Gradients](https://mlsanity.com/paper/1612.03226/) (2016)
- [Measuring the Accuracy of Automatic Speech Recognition Solutions](https://mlsanity.com/paper/2408.16287/) (2024)
- [Bi-Directional Lattice Recurrent Neural Networks for Confidence Estimation](https://mlsanity.com/paper/1810.13024/) (2018)
- [Multi-Task Learning for End-to-End ASR Word and Utterance Confidence with Deletion Prediction](https://mlsanity.com/paper/2104.12870/) (2021)
- [A Corpus for Modeling Word Importance in Spoken Dialogue Transcripts](https://mlsanity.com/paper/1801.09746/) (2018)
- [Social, Environmental, and Technical: Factors at Play in the Current Use and Future Design of Small-Group Captioning](https://mlsanity.com/paper/2109.10412/) (2021)

## Built on by (most cited first, 6 of 6)

- [Data-Efficient ASR Personalization for Non-Normative Speech Using an Uncertainty-Based Phoneme Difficulty Score for Guided Sampling](https://mlsanity.com/paper/2509.20396/) (2025)
- [When Misinformation Speaks and Converses: Rethinking Fact-Checking in Audio Platforms](https://mlsanity.com/paper/2604.16767/) (2026)
- [Demonstration of Adapt4Me: An Uncertainty-Aware Authoring Environment for Personalizing Automatic Speech Recognition to Non-normative Speech](https://mlsanity.com/paper/2603.20112/) (2026)
- [Utterance-Level Methods for Identifying Reliable ASR-Output for Child Speech](https://mlsanity.com/paper/2604.19801/) (2026)
- [Proactive for Uncertainty: Cause-Aware Error Diagnosis and Interactive Clarification for Spoken Dialogue Systems](https://mlsanity.com/paper/2605.25404/) (2026)
- [Transcribing Children's Speech: ASR Performance and Obtaining Reliable Orthographic Transcriptions](https://mlsanity.com/paper/2605.28833/) (2026)

## Abstract (arXiv)

Despite advances in Automatic Speech Recognition (ASR), transcription errors persist and require manual correction. Confidence scores, which indicate the certainty of ASR results, could assist users in identifying and correcting errors. This study evaluates the reliability of confidence scores for error detection through a comprehensive analysis of end-to-end ASR models and a user study with 36 participants. The results show that while confidence scores correlate with transcription accuracy, their error detection performance is limited. Classifiers frequently miss errors or generate many false positives, undermining their practical utility. Confidence-based error detection neither improved correction efficiency nor was perceived as helpful by participants. These findings highlight the limitations of confidence scores and the need for more sophisticated approaches to improve user interaction and explainability of ASR results.