# Improving Confidence Estimation on Out-of-Domain Data for End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/2110.03327 · HTML: https://mlsanity.com/paper/2110.03327/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 19 of 19)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [A Better and Faster End-to-End Model for Streaming ASR](https://mlsanity.com/paper/2011.10798/) (2020)
- [Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability](https://mlsanity.com/paper/2007.15188/) (2020)
- [Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition](https://mlsanity.com/paper/2010.11428/) (2020)
- [On the limit of English conversational speech recognition](https://mlsanity.com/paper/2105.00982/) (2021)
- [Learning Word-Level Confidence For Subword End-to-End ASR](https://mlsanity.com/paper/2103.06716/) (2021)
- [An evaluation of word-level confidence estimation for end-to-end automatic speech recognition](https://mlsanity.com/paper/2101.05525/) (2021)
- [Confidence Estimation and Deletion Prediction Using Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1810.13025/) (2018)
- [Bi-Directional Lattice Recurrent Neural Networks for Confidence Estimation](https://mlsanity.com/paper/1810.13024/) (2018)
- [Residual Energy-Based Models for End-to-End Speech Recognition](https://mlsanity.com/paper/2103.14152/) (2021)
- [Multi-Task Learning for End-to-End ASR Word and Utterance Confidence with Deletion Prediction](https://mlsanity.com/paper/2104.12870/) (2021)
- [Utterance-level neural confidence measure for end-to-end children speech recognition](https://mlsanity.com/paper/2109.07750/) (2021)

## Built on by (most cited first, 7 of 7)

- [A Closer Look at AUROC and AUPRC under Class Imbalance](https://mlsanity.com/paper/2401.06091/) (2024)
- [ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognition](https://mlsanity.com/paper/2305.16065/) (2023)
- [MSAC: Multiple Speech Attribute Control Method for Reliable Speech Emotion Recognition](https://mlsanity.com/paper/2308.04025/) (2023)
- [Adopting Whisper for Confidence Estimation](https://mlsanity.com/paper/2502.13446/) (2025)
- [Exploring Energy-based Language Models with Different Architectures and Training Methods for Speech Recognition](https://mlsanity.com/paper/2305.12676/) (2023)
- [TeLeS: Temporal Lexeme Similarity Score to Estimate Confidence in End-to-End ASR](https://mlsanity.com/paper/2401.03251/) (2024)
- [Accurate and Reliable Confidence Estimation Based on Non-Autoregressive End-to-End Speech Recognition System](https://mlsanity.com/paper/2305.10680/) (2023)

## Abstract (arXiv)

As end-to-end automatic speech recognition (ASR) models reach promising performance, various downstream tasks rely on good confidence estimators for these systems. Recent research has shown that model-based confidence estimators have a significant advantage over using the output softmax probabilities. If the input data to the speech recogniser is from mismatched acoustic and linguistic conditions, the ASR performance and the corresponding confidence estimators may exhibit severe degradation. Since confidence models are often trained on the same in-domain data as the ASR, generalising to out-of-domain (OOD) scenarios is challenging. By keeping the ASR model untouched, this paper proposes two approaches to improve the model-based confidence estimators on OOD data: using pseudo transcriptions and an additional OOD language model. With an ASR model trained on LibriSpeech, experiments show that the proposed methods can greatly improve the confidence metrics on TED-LIUM and Switchboard datasets while preserving in-domain performance. Furthermore, the improved confidence estimators are better calibrated on OOD data and can provide a much more reliable criterion for data selection.