# Context-Aware Selective Label Smoothing for Calibrating Sequence Recognition Model

arXiv: https://arxiv.org/abs/2303.06946 · HTML: https://mlsanity.com/paper/2303.06946/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis](https://mlsanity.com/paper/1904.01906/) (2019)
- [Local Temperature Scaling for Probability Calibration](https://mlsanity.com/paper/2008.05105/) (2020)
- [Uncertainty-based Traffic Accident Anticipation with Spatio-Temporal Relational Learning](https://mlsanity.com/paper/2008.00334/) (2020)
- [Calibration of Encoder Decoder Models for Neural Machine Translation](https://mlsanity.com/paper/1903.00802/) (2019)
- [Bin-wise Temperature Scaling (BTS): Improvement in Confidence Calibration Performance through Simple Scaling Techniques](https://mlsanity.com/paper/1908.11528/) (2019)
- [Semantic Label Smoothing for Sequence to Sequence Problems](https://mlsanity.com/paper/2010.07447/) (2020)
- [On Calibration of Scene-Text Recognition Models](https://mlsanity.com/paper/2012.12643/) (2020)
- [Token-level and sequence-level loss smoothing for RNN language models](https://mlsanity.com/paper/1805.05062/) (2018)
- [Temporal Probability Calibration](https://mlsanity.com/paper/2002.02644/) (2020)
- [Homophone-based Label Smoothing in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2004.03437/) (2020)

## Built on by (most cited first, 5 of 5)

- [One-Shot Diffusion Mimicker for Handwritten Text Generation](https://mlsanity.com/paper/2409.04004/) (2024)
- [Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimization](https://mlsanity.com/paper/2407.02243/) (2024)
- [Beyond Isolated Words: Diffusion Brush for Handwritten Text-Line Generation](https://mlsanity.com/paper/2508.03256/) (2025)
- [Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models](https://mlsanity.com/paper/2405.14161/) (2024)
- [Perception and Semantic Aware Regularization for Sequential Confidence Calibration](https://mlsanity.com/paper/2305.19498/) (2023)

## Abstract (arXiv)

Despite the success of deep neural network (DNN) on sequential data (i.e., scene text and speech) recognition, it suffers from the over-confidence problem mainly due to overfitting in training with the cross-entropy loss, which may make the decision-making less reliable. Confidence calibration has been recently proposed as one effective solution to this problem. Nevertheless, the majority of existing confidence calibration methods aims at non-sequential data, which is limited if directly applied to sequential data since the intrinsic contextual dependency in sequences or the class-specific statistical prior is seldom exploited. To the end, we propose a Context-Aware Selective Label Smoothing (CASLS) method for calibrating sequential data. The proposed CASLS fully leverages the contextual dependency in sequences to construct confusion matrices of contextual prediction statistics over different classes. Class-specific error rates are then used to adjust the weights of smoothing strength in order to achieve adaptive calibration. Experimental results on sequence recognition tasks, including scene text recognition and speech recognition, demonstrate that our method can achieve the state-of-the-art performance.