# TeLeS: Temporal Lexeme Similarity Score to Estimate Confidence in End-to-End ASR

arXiv: https://arxiv.org/abs/2401.03251 · HTML: https://mlsanity.com/paper/2401.03251/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 6 Jan 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages](https://mlsanity.com/paper/2208.11761/) (2022)
- [Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition](https://mlsanity.com/paper/2010.11428/) (2020)
- [Semi-supervised ASR by End-to-end Self-training](https://mlsanity.com/paper/2001.09128/) (2020)
- [Learning Word-Level Confidence For Subword End-to-End ASR](https://mlsanity.com/paper/2103.06716/) (2021)
- [Towards Lifelong Learning of End-to-end ASR](https://mlsanity.com/paper/2104.01616/) (2021)
- [An evaluation of word-level confidence estimation for end-to-end automatic speech recognition](https://mlsanity.com/paper/2101.05525/) (2021)
- [ASR Rescoring and Confidence Estimation with ELECTRA](https://mlsanity.com/paper/2110.01857/) (2021)
- [Bi-Directional Lattice Recurrent Neural Networks for Confidence Estimation](https://mlsanity.com/paper/1810.13024/) (2018)
- [Subword Dictionary Learning and Segmentation Techniques for Automatic Speech Recognition in Tamil and Kannada](https://mlsanity.com/paper/2207.13331/) (2022)
- [Residual Energy-Based Models for End-to-End Speech Recognition](https://mlsanity.com/paper/2103.14152/) (2021)
- [Multi-Task Learning for End-to-End ASR Word and Utterance Confidence with Deletion Prediction](https://mlsanity.com/paper/2104.12870/) (2021)
- [Improving Confidence Estimation on Out-of-Domain Data for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.03327/) (2021)
- [Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2212.08703/) (2022)
- [On Modeling ASR Word Confidence](https://mlsanity.com/paper/1907.09636/) (2019)
- [Knowledge-driven Subword Grammar Modeling for Automatic Speech Recognition in Tamil and Kannada](https://mlsanity.com/paper/2207.13333/) (2022)
- [Confidence Estimation for Black Box Automatic Speech Recognition Systems Using Lattice Recurrent Neural Networks](https://mlsanity.com/paper/1910.11933/) (2019)
- [Word-level confidence estimation for RNN transducers](https://mlsanity.com/paper/2110.15222/) (2021)
- [BLSTM-Based Confidence Estimation for End-to-End Speech Recognition](https://mlsanity.com/paper/2312.14609/) (2023)

## Built on by (most cited first, 1 of 1)

- [Enhancing ASR Performance in the Medical Domain for Dravidian Languages](https://mlsanity.com/paper/2604.19797/) (2026)

## Abstract (arXiv)

Confidence estimation of predictions from an End-to-End (E2E) Automatic Speech Recognition (ASR) model benefits ASR's downstream and upstream tasks. Class-probability-based confidence scores do not accurately represent the quality of overconfident ASR predictions. An ancillary Confidence Estimation Model (CEM) calibrates the predictions. State-of-the-art (SOTA) solutions use binary target scores for CEM training. However, the binary labels do not reveal the granular information of predicted words, such as temporal alignment between reference and hypothesis and whether the predicted word is entirely incorrect or contains spelling errors. Addressing this issue, we propose a novel Temporal-Lexeme Similarity (TeLeS) confidence score to train CEM. To address the data imbalance of target scores while training CEM, we use shrinkage loss to focus on hard-to-learn data points and minimise the impact of easily learned data points. We conduct experiments with ASR models trained in three languages, namely Hindi, Tamil, and Kannada, with varying training data sizes. Experiments show that TeLeS generalises well across domains. To demonstrate the applicability of the proposed method, we formulate a TeLeS-based Acquisition (TeLeS-A) function for sampling uncertainty in active learning. We observe a significant reduction in the Word Error Rate (WER) as compared to SOTA methods.