mlsanityData to 14 Oct 2026

Analyzing the Importance of Blank for CTC-Based Knowledge Distillation

arXiv 2506.01503 ↗First posted 2 Jun 2025cs.LG cs.SD eess.AScited by 0

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

With the rise of large pre-trained foundation models for automatic speech recognition new challenges appear. While the performance of these models is good, runtime and cost of inference increases. One approach to make use of their strength while retaining efficiency is to distill their knowledge to smaller models during training. In this work, we explore different CTC-based distillation variants, focusing on blank token handling. We show that common approaches like blank elimination do not always work off the shelf. We explore new blank selection patterns as a potential sweet spot between standard knowledge distillation and blank elimination mechanisms. Through the introduction of a symmetric selection method, we are able to remove the CTC loss during knowledge distillation with minimal to no performance degradation. With this, we make the training independent from target labels, potentially allowing for distillation on untranscribed audio data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 16 of its 32 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

16 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Robust Knowledge Distillation from RNN-T Models With Noisy Training Labels Using Full-Sum Loss10 Mar
  2. 2022Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR11 Oct

Other references · 14

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2018Self-Attention with Relative Position Representationscited by 1,120
  8. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  9. 2020Efficient Knowledge Distillation for RNN-Transducer Modelscited by 38
  10. 2021Why does CTC result in peaky behavior?cited by 33
  11. 2022Flashlight: Enabling Innovation in Tools for Machine Learningcited by 25
  12. 2019Guiding CTC Posterior Spike Timings for Improved Posterior Fusion and Knowledge Distillationcited by 18
2 more
  1. 2016RETURNN: The RWTH Extensible Training framework for Universal Recurrent Neural Networkscited by 16
  2. 2024Less Peaky and More Accurate CTC Forced Alignment by Label Priorscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data
  2. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  3. 2022Inter-KD: Intermediate Knowledge Distillation for CTC-Based Automatic Speech Recognition
  4. 2023Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition
  5. 2021Knowledge Distillation Leveraging Alternative Soft Targets from Non-Parallel Qualified Speech Data
  6. 2024CTC Blank Triggered Dynamic Layer-Skipping for Efficient CTC-based Speech Recognition
  7. 2026ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition
  8. 2023Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation