# Multi-Task Learning for End-to-End ASR Word and Utterance Confidence with Deletion Prediction

arXiv: https://arxiv.org/abs/2104.12870 · HTML: https://mlsanity.com/paper/2104.12870/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 44 references that are in the map.
- First posted on arXiv 26 Apr 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling](https://mlsanity.com/paper/1902.08295/) (2019)
- [Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition](https://mlsanity.com/paper/2010.11428/) (2020)
- [Learning Word-Level Confidence For Subword End-to-End ASR](https://mlsanity.com/paper/2103.06716/) (2021)
- [Confidence Estimation and Deletion Prediction Using Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1810.13025/) (2018)
- [Bi-Directional Lattice Recurrent Neural Networks for Confidence Estimation](https://mlsanity.com/paper/1810.13024/) (2018)
- [Parallel Rescoring with Transformer for Streaming On-Device Speech Recognition](https://mlsanity.com/paper/2008.13093/) (2020)
- [Residual Energy-Based Models for End-to-End Speech Recognition](https://mlsanity.com/paper/2103.14152/) (2021)

## Built on by (most cited first, 8 of 8)

- [Improving Confidence Estimation on Out-of-Domain Data for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.03327/) (2021)
- [Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2212.08703/) (2022)
- [Evaluating ASR Confidence Scores for Automated Error Detection in User-Assisted Correction Interfaces](https://mlsanity.com/paper/2503.15124/) (2025)
- [MSAC: Multiple Speech Attribute Control Method for Reliable Speech Emotion Recognition](https://mlsanity.com/paper/2308.04025/) (2023)
- [Word-level confidence estimation for RNN transducers](https://mlsanity.com/paper/2110.15222/) (2021)
- [Adopting Whisper for Confidence Estimation](https://mlsanity.com/paper/2502.13446/) (2025)
- [TeLeS: Temporal Lexeme Similarity Score to Estimate Confidence in End-to-End ASR](https://mlsanity.com/paper/2401.03251/) (2024)
- [Accurate and Reliable Confidence Estimation Based on Non-Autoregressive End-to-End Speech Recognition System](https://mlsanity.com/paper/2305.10680/) (2023)

## Abstract (arXiv)

Confidence scores are very useful for downstream applications of automatic speech recognition (ASR) systems. Recent works have proposed using neural networks to learn word or utterance confidence scores for end-to-end ASR. In those studies, word confidence by itself does not model deletions, and utterance confidence does not take advantage of word-level training signals. This paper proposes to jointly learn word confidence, word deletion, and utterance confidence. Empirical results show that multi-task learning with all three objectives improves confidence metrics (NCE, AUC, RMSE) without the need for increasing the model size of the confidence estimation module. Using the utterance-level confidence for rescoring also decreases the word error rates on Google's Voice Search and Long-tail Maps datasets by 3-5% relative, without needing a dedicated neural rescorer.