# TutorNet: Towards Flexible Knowledge Distillation for End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/2008.00671 · HTML: https://mlsanity.com/paper/2008.00671/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Aug 2020.

## Built on by (most cited first, 12 of 12)

- [Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher](https://mlsanity.com/paper/2110.08532/) (2021)
- [Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models](https://mlsanity.com/paper/2110.03334/) (2021)
- [Inter-KD: Intermediate Knowledge Distillation for CTC-Based Automatic Speech Recognition](https://mlsanity.com/paper/2211.15075/) (2022)
- [Large-Scale Data-Free Knowledge Distillation for ImageNet via Multi-Resolution Data Generation](https://mlsanity.com/paper/2411.17046/) (2024)
- [Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation](https://mlsanity.com/paper/2103.09903/) (2021)
- [Prototype-guided Cross-task Knowledge Distillation for Large-scale Models](https://mlsanity.com/paper/2212.13180/) (2022)
- [Guiding Frame-Level CTC Alignments Using Self-knowledge Distillation](https://mlsanity.com/paper/2406.07909/) (2024)
- [EM-Network: Oracle Guided Self-distillation for Sequence Learning](https://mlsanity.com/paper/2306.10058/) (2023)
- [An Effective Training Framework for Light-Weight Automatic Speech Recognition Models](https://mlsanity.com/paper/2505.16991/) (2025)
- [Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Models](https://mlsanity.com/paper/2111.03664/) (2021)
- [Speaker-Adapted End-to-End Visual Speech Recognition for Continuous Spanish](https://mlsanity.com/paper/2311.12480/) (2023)
- [Distillation-based Layer Dropping (DLD): Effective End-to-end Framework for Dynamic Speech Networks](https://mlsanity.com/paper/2601.16117/) (2026)

## Abstract (arXiv)

In recent years, there has been a great deal of research in developing end-to-end speech recognition models, which enable simplifying the traditional pipeline and achieving promising results. Despite their remarkable performance improvements, end-to-end models typically require expensive computational cost to show successful performance. To reduce this computational burden, knowledge distillation (KD), which is a popular model compression method, has been used to transfer knowledge from a deep and complex model (teacher) to a shallower and simpler model (student). Previous KD approaches have commonly designed the architecture of the student model by reducing the width per layer or the number of layers of the teacher model. This structural reduction scheme might limit the flexibility of model selection since the student model structure should be similar to that of the given teacher. To cope with this limitation, we propose a new KD method for end-to-end speech recognition, namely TutorNet, that can transfer knowledge across different types of neural networks at the hidden representation-level as well as the output-level. For concrete realizations, we firstly apply representation-level knowledge distillation (RKD) during the initialization step, and then apply the softmax-level knowledge distillation (SKD) combined with the original task learning. When the student is trained with RKD, we make use of frame weighting that points out the frames to which the teacher model pays more attention. Through a number of experiments on LibriSpeech dataset, it is verified that the proposed method not only distills the knowledge between networks with different topologies but also significantly contributes to improving the word error rate (WER) performance of the distilled student. Interestingly, TutorNet allows the student model to surpass its teacher's performance in some particular cases.