mlsanityData to 7 Oct 2026

Learning Word-Level Confidence For Subword End-to-End ASR

arXiv 2103.06716 ↗First posted 11 Mar 2021eess.AS cs.CL cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

We study the problem of word-level confidence estimation in subword-based end-to-end (E2E) models for automatic speech recognition (ASR). Although prior works have proposed training auxiliary confidence models for ASR systems, they do not extend naturally to systems that operate on word-pieces (WP) as their vocabulary. In particular, ground truth WP correctness labels are needed for training confidence models, but the non-unique tokenization from word to WP causes inaccurate labels to be generated. This paper proposes and studies two confidence models of increasing complexity to solve this problem. The final model uses self-attention to directly learn word-level confidence without needing subword tokenization, and exploits full context features from multiple hypotheses to improve confidence accuracy. Experiments on Voice Search and long-tail test sets show standard metrics (e.g., NCE, AUC, RMSE) improving substantially. The proposed confidence module also enables a model selection approach to combine an on-device E2E model with a hybrid model on the server to address the rare word recognition problem for the E2E model.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 21 of its 40 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

21 of 40 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  6. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  7. 2015Attention-Based Models for Speech Recognitioncited by 400
  8. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  9. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  10. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  11. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  12. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
9 more
  1. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
  2. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  3. 2019Transformer-Transducer: End-to-End Speech Recognition with Self-Attentioncited by 68
  4. 2020Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capabilitycited by 63
  5. 2020Deliberation Model Based Two-Pass End-to-End Speech Recognitioncited by 57
  6. 2020Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognitioncited by 35
  7. 2018Confidence Estimation and Deletion Prediction Using Bidirectional Recurrent Neural Networkscited by 12
  8. 2018Bi-Directional Lattice Recurrent Neural Networks for Confidence Estimationcited by 11
  9. 2020Parallel Rescoring with Transformer for Streaming On-Device Speech Recognitioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 4; 2023: 3; 2024: 2; 2025: 3; 2026: 1.

22 papers.

Sort:

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  2. 2021Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learningcited by 26
  3. 2022Pseudo Label Is Better Than Human Labelcited by 25
  4. 2021On Addressing Practical Challenges for RNN-Transducercited by 19
  5. 2021ASR Rescoring and Confidence Estimation with ELECTRAcited by 16
  6. 2023Multi-stage Large Language Model Correction for Speech Recognitioncited by 13
  7. 2021Understanding Medical Conversations: Rich Transcription, Confidence Scores & Information Extractioncited by 10
  8. 2021Residual Energy-Based Models for End-to-End Speech Recognitioncited by 9
  9. 2021Multi-Task Learning for End-to-End ASR Word and Utterance Confidence with Deletion Predictioncited by 8
  10. 2022RED-ACE: Robust Error Detection for ASR using Confidence Embeddingscited by 8
  11. 2021Improving Confidence Estimation on Out-of-Domain Data for End-to-End Speech Recognitioncited by 7
  12. 2022Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognitioncited by 7
10 more
  1. 2023ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognitioncited by 7
  2. 2022Cross-Modal ASR Post-Processing System for Error Correction and Utterance Rejectioncited by 6
  3. 2023NoRefER: a Referenceless Quality Metric for Automatic Speech Recognition via Semi-Supervised Language Model Fine-Tuning with Contrastive Learningcited by 4
  4. 2021Word-level confidence estimation for RNN transducerscited by 3
  5. 2024Error Correction by Paying Attention to Both Acoustic and Confidence References for Automatic Speech Recognitioncited by 2
  6. 2025On the Robust Approximation of ASR Metricscited by 2
  7. 2025Adopting Whisper for Confidence Estimationcited by 2
  8. 2024TeLeS: Temporal Lexeme Similarity Score to Estimate Confidence in End-to-End ASRcited by 1
  9. 2025Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypothesescited by 1
  10. 2026Proactive for Uncertainty: Cause-Aware Error Diagnosis and Interactive Clarification for Spoken Dialogue Systemscited by 0