# Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition

arXiv: https://arxiv.org/abs/2010.11428 · HTML: https://mlsanity.com/paper/2010.11428/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2020.

## Built on by (most cited first, 30 of 35)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio](https://mlsanity.com/paper/2501.11378/) (2025)
- [Learning Word-Level Confidence For Subword End-to-End ASR](https://mlsanity.com/paper/2103.06716/) (2021)
- [ASR Rescoring and Confidence Estimation with ELECTRA](https://mlsanity.com/paper/2110.01857/) (2021)
- [Multi-stage Large Language Model Correction for Speech Recognition](https://mlsanity.com/paper/2310.11532/) (2023)
- [Boosting Cross-Domain Speech Recognition with Self-Supervision](https://mlsanity.com/paper/2206.09783/) (2022)
- [Residual Energy-Based Models for End-to-End Speech Recognition](https://mlsanity.com/paper/2103.14152/) (2021)
- [Multi-Task Learning for End-to-End ASR Word and Utterance Confidence with Deletion Prediction](https://mlsanity.com/paper/2104.12870/) (2021)
- [RED-ACE: Robust Error Detection for ASR using Confidence Embeddings](https://mlsanity.com/paper/2203.07172/) (2022)
- [Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models](https://mlsanity.com/paper/2405.14161/) (2024)
- [Improving Confidence Estimation on Out-of-Domain Data for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.03327/) (2021)
- [Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2212.08703/) (2022)
- [ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognition](https://mlsanity.com/paper/2305.16065/) (2023)
- [Cross-Modal ASR Post-Processing System for Error Correction and Utterance Rejection](https://mlsanity.com/paper/2201.03313/) (2022)
- [Alternative Pseudo-Labeling for Semi-Supervised Automatic Speech Recognition](https://mlsanity.com/paper/2308.06547/) (2023)
- [Evaluating ASR Confidence Scores for Automated Error Detection in User-Assisted Correction Interfaces](https://mlsanity.com/paper/2503.15124/) (2025)
- [Relaxed Attention: A Simple Method to Boost Performance of End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2107.01275/) (2021)
- [Confidence-based Ensembles of End-to-End Speech Recognition Models](https://mlsanity.com/paper/2306.15824/) (2023)
- [Are Generative Models Underconfident? Better Quality Estimation with Boosted Model Probability](https://mlsanity.com/paper/2502.11115/) (2025)
- [Word-level confidence estimation for RNN transducers](https://mlsanity.com/paper/2110.15222/) (2021)
- [Who Needs Decoders? Efficient Estimation of Sequence-level Attributes](https://mlsanity.com/paper/2305.05098/) (2023)
- [Automatic Speech Recognition System-Independent Word Error Rate Estimation](https://mlsanity.com/paper/2404.16743/) (2024)
- [Error Correction by Paying Attention to Both Acoustic and Confidence References for Automatic Speech Recognition](https://mlsanity.com/paper/2407.12817/) (2024)
- [Confidence Estimation for Automatic Detection of Depression and Alzheimer's Disease Based on Clinical Interviews](https://mlsanity.com/paper/2407.19984/) (2024)
- [Adopting Whisper for Confidence Estimation](https://mlsanity.com/paper/2502.13446/) (2025)
- [Perception and Semantic Aware Regularization for Sequential Confidence Calibration](https://mlsanity.com/paper/2305.19498/) (2023)
- [TeLeS: Temporal Lexeme Similarity Score to Estimate Confidence in End-to-End ASR](https://mlsanity.com/paper/2401.03251/) (2024)
- [Customizing Speech Recognition Model with Large Language Model Feedback](https://mlsanity.com/paper/2506.11091/) (2025)
- [Data-Efficient ASR Personalization for Non-Normative Speech Using an Uncertainty-Based Phoneme Difficulty Score for Guided Sampling](https://mlsanity.com/paper/2509.20396/) (2025)
- [From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection](https://mlsanity.com/paper/2606.23060/) (2026)

## Abstract (arXiv)

For various speech-related tasks, confidence scores from a speech recogniser are a useful measure to assess the quality of transcriptions. In traditional hidden Markov model-based automatic speech recognition (ASR) systems, confidence scores can be reliably obtained from word posteriors in decoding lattices. However, for an ASR system with an auto-regressive decoder, such as an attention-based sequence-to-sequence model, computing word posteriors is difficult. An obvious alternative is to use the decoder softmax probability as the model confidence. In this paper, we first examine how some commonly used regularisation methods influence the softmax-based confidence scores and study the overconfident behaviour of end-to-end models. Then we propose a lightweight and effective approach named confidence estimation module (CEM) on top of an existing end-to-end ASR model. Experiments on LibriSpeech show that CEM can mitigate the overconfidence problem and can produce more reliable confidence scores with and without shallow fusion of a language model. Further analysis shows that CEM generalises well to speech from a moderately mismatched domain and can potentially improve downstream tasks such as semi-supervised learning.