# Self-distillation Regularized Connectionist Temporal Classification Loss for Text Recognition: A Simple Yet Effective Approach arXiv: https://arxiv.org/abs/2308.08806 ยท HTML: https://mlsanity.com/paper/2308.08806/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. - Builds on 47 of its 82 references that are in the map. - First posted on arXiv 17 Aug 2023. ## Placed in - Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/) ## Builds on (most cited first, 30 of 47) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017) - [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016) - [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015) - [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016) - [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021) - [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014) - [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021) - [What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis](https://mlsanity.com/paper/1904.01906/) (2019) - [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://mlsanity.com/paper/2207.06966/) (2022) - [Towards Accurate Scene Text Recognition with Semantic Reasoning Networks](https://mlsanity.com/paper/2003.12294/) (2020) - [Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition](https://mlsanity.com/paper/1811.00751/) (2018) - [Robust Scene Text Recognition with Automatic Rectification](https://mlsanity.com/paper/1603.03915/) (2016) - [SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://mlsanity.com/paper/2005.10977/) (2020) - [Scene Text Detection and Recognition: The Deep Learning Era](https://mlsanity.com/paper/1811.04256/) (2018) - [Focusing Attention: Towards Accurate Text Recognition in Natural Images](https://mlsanity.com/paper/1709.02054/) (2017) - [PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR System](https://mlsanity.com/paper/2206.03001/) (2022) - [From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network](https://mlsanity.com/paper/2108.09661/) (2021) - [SVTR: Scene Text Recognition with a Single Visual Model](https://mlsanity.com/paper/2205.00159/) (2022) - [EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding](https://mlsanity.com/paper/1507.08240/) (2015) - [NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition](https://mlsanity.com/paper/1806.00926/) (2018) - [Sequence-to-Sequence Contrastive Learning for Text Recognition](https://mlsanity.com/paper/2012.10873/) (2020) - [MASTER: Multi-Aspect Non-local Network for Scene Text Recognition](https://mlsanity.com/paper/1910.02562/) (2019) - [What If We Only Use Real Datasets for Scene Text Recognition? Toward Scene Text Recognition With Fewer Labels](https://mlsanity.com/paper/2103.04400/) (2021) - [Scene Text Recognition from Two-Dimensional Perspective](https://mlsanity.com/paper/1809.06508/) (2018) - [Benchmarking Chinese Text Recognition: Datasets, Baselines, and an Empirical Study](https://mlsanity.com/paper/2112.15093/) (2021) - [GTC: Guided Training of CTC Towards Efficient and Accurate Scene Text Recognition](https://mlsanity.com/paper/2002.01276/) (2020) - [Reading and Writing: Discriminative and Generative Modeling for Self-Supervised Text Recognition](https://mlsanity.com/paper/2207.00193/) (2022) ## Built on by (most cited first, 8 of 8) - [SVTRv2: CTC Beats Encoder-Decoder Models in Scene Text Recognition](https://mlsanity.com/paper/2411.15858/) (2024) - [Instruction-Guided Scene Text Recognition](https://mlsanity.com/paper/2401.17851/) (2024) - [Out of Length Text Recognition with Sub-String Matching](https://mlsanity.com/paper/2407.12317/) (2024) - [TextSSR: Diffusion-based Data Synthesis for Scene Text Recognition](https://mlsanity.com/paper/2412.01137/) (2024) - [Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition](https://mlsanity.com/paper/2503.18746/) (2025) - [SVIPTR: Fast and Efficient Scene Text Recognition with Vision Permutable Extractor](https://mlsanity.com/paper/2401.10110/) (2024) - [LRANet++: Low-Rank Approximation Network for Accurate and Efficient Text Spotting](https://mlsanity.com/paper/2511.05818/) (2025) - [MDiff4STR: Mask Diffusion Model for Scene Text Recognition](https://mlsanity.com/paper/2512.01422/) (2025) ## Abstract (arXiv) Text recognition methods are gaining rapid development. Some advanced techniques, e.g., powerful modules, language models, and un- and semi-supervised learning schemes, consecutively push the performance on public benchmarks forward. However, the problem of how to better optimize a text recognition model from the perspective of loss functions is largely overlooked. CTC-based methods, widely used in practice due to their good balance between performance and inference speed, still grapple with accuracy degradation. This is because CTC loss emphasizes the optimization of the entire sequence target while neglecting to learn individual characters. We propose a self-distillation scheme for CTC-based model to address this issue. It incorporates a framewise regularization term in CTC loss to emphasize individual supervision, and leverages the maximizing-a-posteriori of latent alignment to solve the inconsistency problem that arises in distillation between CTC-based models. We refer to the regularized CTC loss as Distillation Connectionist Temporal Classification (DCTC) loss. DCTC loss is module-free, requiring no extra parameters, longer inference lag, or additional training data or phases. Extensive experiments on public benchmarks demonstrate that DCTC can boost text recognition model accuracy by up to 2.6%, without any of these drawbacks.