mlsanityData to 14 Oct 2026

Guiding Frame-Level CTC Alignments Using Self-knowledge Distillation

arXiv 2406.07909 ↗First posted 12 Jun 2024eess.AS cs.CL cs.SDcited by 2

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Transformer encoder with connectionist temporal classification (CTC) framework is widely used for automatic speech recognition (ASR). However, knowledge distillation (KD) for ASR displays a problem of disagreement between teacher-student models in frame-level alignment which ultimately hinders it from improving the student model's performance. In order to resolve this problem, this paper introduces a self-knowledge distillation (SKD) method that guides the frame-level alignment during the training time. In contrast to the conventional method using separate teacher and student models, this study introduces a simple and effective method sharing encoder layers and applying the sub-model as the student model. Overall, our approach is effective in improving both the resource efficiency as well as performance. We also conducted an experimental analysis of the spike timings to illustrate that the proposed method improves performance by reducing the alignment disagreement.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 20 of its 30 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

20 of 30 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Inter-KD: Intermediate Knowledge Distillation for CTC-Based Automatic Speech Recognition28 Nov

Other references · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  8. 2015Listen, Attend and Spellcited by 543
  9. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  10. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  11. 2018Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducercited by 96
  12. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 96
7 more
  1. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
  2. 2019Guiding CTC Posterior Spike Timings for Improved Posterior Fusion and Knowledge Distillationcited by 18
  3. 2021Layer Pruning on Demand with Intermediate CTCcited by 13
  4. 2022Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence taskscited by 13
  5. 2020TutorNet: Towards Flexible Knowledge Distillation for End-to-End Speech Recognitioncited by 12
  6. 2022Blank Collapse: Compressing CTC emission for the faster decodingcited by 3
  7. 2023Key Frame Mechanism For Efficient Conformer Based End-to-end Speech Recognitioncited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024CR-CTC: Consistency regularization on CTC for improved speech recognitioncited by 17
  2. 2025Delayed-KD: Delayed Knowledge Distillation based CTC for Low-Latency Streaming ASRcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition
  2. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  3. 2026Alignment-Path Distillation from Non-streaming ASR-LLMs for Streaming Speech Recognition
  4. 2021Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation
  5. 2023Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition
  6. 2021Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Models
  7. 2026ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition
  8. 2023Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer