Adopting Whisper for Confidence Estimation
Not on the map. Why?
Abstract (arXiv)
Recent research on word-level confidence estimation for speech recognition systems has primarily focused on lightweight models known as Confidence Estimation Modules (CEMs), which rely on hand-engineered features derived from Automatic Speech Recognition (ASR) outputs. In contrast, we propose a novel end-to-end approach that leverages the ASR model itself (Whisper) to generate word-level confidence scores. Specifically, we introduce a method in which the Whisper model is fine-tuned to produce scalar confidence scores given an audio input and its corresponding hypothesis transcript. Our experiments demonstrate that the fine-tuned Whisper-tiny model, comparable in size to a strong CEM baseline, achieves similar performance on the in-domain dataset and surpasses the CEM baseline on eight out-of-domain datasets, whereas the fine-tuned Whisper-large model consistently outperforms the CEM baseline by a substantial margin across all datasets.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 35 references that are in the map. First posted on arXiv 19 Feb 2025.
Builds on
23 of 35 references.
References · 23
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
- 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
- 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
- 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
- 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
- 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
11 more
- 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
- 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
- 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
- 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
- 2020Deliberation Model Based Two-Pass End-to-End Speech Recognitioncited by 57
- 2020Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognitioncited by 35
- 2021Learning Word-Level Confidence For Subword End-to-End ASRcited by 22
- 2018Confidence Estimation and Deletion Prediction Using Bidirectional Recurrent Neural Networkscited by 12
- 2021Multi-Task Learning for End-to-End ASR Word and Utterance Confidence with Deletion Predictioncited by 8
- 2021Improving Confidence Estimation on Out-of-Domain Data for End-to-End Speech Recognitioncited by 7
- 2022Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognitioncited by 7
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.