mlsanityData to 7 Oct 2026

Multi-Task Learning for End-to-End ASR Word and Utterance Confidence with Deletion Prediction

arXiv 2104.12870 ↗First posted 26 Apr 2021eess.AS cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Confidence scores are very useful for downstream applications of automatic speech recognition (ASR) systems. Recent works have proposed using neural networks to learn word or utterance confidence scores for end-to-end ASR. In those studies, word confidence by itself does not model deletions, and utterance confidence does not take advantage of word-level training signals. This paper proposes to jointly learn word confidence, word deletion, and utterance confidence. Empirical results show that multi-task learning with all three objectives improves confidence metrics (NCE, AUC, RMSE) without the need for increasing the model size of the confidence estimation module. Using the utterance-level confidence for rescoring also decreases the word error rates on Google's Voice Search and Long-tail Maps datasets by 3-5% relative, without needing a dedicated neural rescorer.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 11 of its 44 references that are in the map. First posted on arXiv 26 Apr 2021.

Builds on

11 of 44 references.

References · 11

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  4. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
  5. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  6. 2020Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognitioncited by 35
  7. 2021Learning Word-Level Confidence For Subword End-to-End ASRcited by 22
  8. 2018Confidence Estimation and Deletion Prediction Using Bidirectional Recurrent Neural Networkscited by 12
  9. 2018Bi-Directional Lattice Recurrent Neural Networks for Confidence Estimationcited by 11
  10. 2020Parallel Rescoring with Transformer for Streaming On-Device Speech Recognitioncited by 9
  11. 2021Residual Energy-Based Models for End-to-End Speech Recognitioncited by 9

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 1; 2023: 2; 2024: 1; 2025: 2.

8 papers.

  1. 2021Improving Confidence Estimation on Out-of-Domain Data for End-to-End Speech Recognitioncited by 7
  2. 2022Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognitioncited by 7
  3. 2025Evaluating ASR Confidence Scores for Automated Error Detection in User-Assisted Correction Interfacescited by 6
  4. 2023MSAC: Multiple Speech Attribute Control Method for Reliable Speech Emotion Recognitioncited by 4
  5. 2021Word-level confidence estimation for RNN transducerscited by 3
  6. 2025Adopting Whisper for Confidence Estimationcited by 2
  7. 2024TeLeS: Temporal Lexeme Similarity Score to Estimate Confidence in End-to-End ASRcited by 1
  8. 2023Accurate and Reliable Confidence Estimation Based on Non-Autoregressive End-to-End Speech Recognition Systemcited by 0