mlsanityData to 14 Oct 2026

An evaluation of word-level confidence estimation for end-to-end automatic speech recognition

arXiv 2101.05525 ↗First posted 14 Jan 2021eess.AS cs.CL cs.SDcited by 18

Not on the map. Why?

Abstract (arXiv)

Quantifying the confidence (or conversely the uncertainty) of a prediction is a highly desirable trait of an automatic system, as it improves the robustness and usefulness in downstream tasks. In this paper we investigate confidence estimation for end-to-end automatic speech recognition (ASR). Previous work has addressed confidence measures for lattice-based ASR, while current machine learning research mostly focuses on confidence measures for unstructured deep learning. However, as the ASR systems are increasingly being built upon deep end-to-end methods, there is little work that tries to develop confidence measures in this context. We fill this gap by providing an extensive benchmark of popular confidence methods on four well-known speech datasets. There are two challenges we overcome in adapting existing methods: working on structured data (sequences) and obtaining confidences at a coarser level than the predictions (words instead of tokens). Our results suggest that a strong baseline can be obtained by scaling the logits by a learnt temperature, followed by estimating the confidence as the negative entropy of the predictive distribution and, finally, sum pooling to aggregate at word level.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 26 of its 47 references that are in the map. First posted on arXiv 14 Jan 2021.

Builds on

26 of 47 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  9. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  11. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  12. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
14 more
  1. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  2. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  3. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
  4. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 211
  5. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 210
  6. 2019Addressing Failure Prediction by Learning Model Confidencecited by 186
  7. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 151
  8. 2019RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentationcited by 63
  9. 2019Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Modelscited by 25
  10. 2020Uncertainty Estimation in Autoregressive Structured Predictioncited by 23
  11. 2018Confidence Scoring Using Whitebox Meta-models with Linear Classifier Probescited by 21
  12. 2017Neural Lattice-to-Sequence Models for Uncertain Inputscited by 20
  13. 2018Bi-Directional Lattice Recurrent Neural Networks for Confidence Estimationcited by 11
  14. 2019On using 2D sequence-to-sequence models for speech recognitioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 3; 2023: 1; 2024: 1; 2025: 1; 2026: 7.

18 papers.

Sort:

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  2. 2021On Addressing Practical Challenges for RNN-Transducercited by 19
  3. 2021ASR Rescoring and Confidence Estimation with ELECTRAcited by 16
  4. 2021Residual Energy-Based Models for End-to-End Speech Recognitioncited by 9
  5. 2021Improving Confidence Estimation on Out-of-Domain Data for End-to-End Speech Recognitioncited by 7
  6. 2022Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognitioncited by 7
  7. 2022Cross-Modal ASR Post-Processing System for Error Correction and Utterance Rejectioncited by 6
  8. 2022How Does Beam Search improve Span-Level Confidence Estimation in Generative Sequence Labeling?cited by 1
  9. 2024TeLeS: Temporal Lexeme Similarity Score to Estimate Confidence in End-to-End ASRcited by 1
  10. 2023Hystoc: Obtaining word confidences for fusion of end-to-end ASR systemscited by 0
  11. 2025UAKNN: Label Distribution Learning via Uncertainty-Aware KNNcited by 0
  12. 2026Utterance-Level Methods for Identifying Reliable ASR-Output for Child Speechcited by 0
6 more
  1. 2026RAS: a Reliability Oriented Metric for Automatic Speech Recognitioncited by 0
  2. 2026Proactive for Uncertainty: Cause-Aware Error Diagnosis and Interactive Clarification for Spoken Dialogue Systemscited by 0
  3. 2026Transcribing Children's Speech: ASR Performance and Obtaining Reliable Orthographic Transcriptionscited by 0
  4. 2026Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancycited by 0
  5. 2026Auditing generative audio calls for known-task audio-llm evaluationcited by 0
  6. 2026Broadening Uncertainty Estimation for Audio Question Answering Across Methods, Formats, and Inputscited by 0