mlsanityData to 14 Oct 2026

Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer

arXiv 2308.16415 ↗First posted 31 Aug 2023cs.CL eess.AScited by 2

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Streaming automatic speech recognition (ASR) models are restricted from accessing future context, which results in worse performance compared to the non-streaming models. To improve the performance of streaming ASR, knowledge distillation (KD) from the non-streaming to streaming model has been studied, mainly focusing on aligning the output token probabilities. In this paper, we propose a layer-to-layer KD from the teacher encoder to the student encoder. To ensure that features are extracted using the same context, we insert auxiliary non-streaming branches to the student and perform KD from the non-streaming teacher layer to the non-streaming auxiliary layer. We design a special KD loss that leverages the autoregressive predictive coding (APC) mechanism to encourage the streaming model to predict unseen future contexts. Experimental results show that the proposed method can significantly reduce the word error rate compared to previous token probability distillation methods.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 24 of its 34 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

24 of 34 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition1 Oct
  2. 2022Conformer with dual-mode chunked attention for joint online and offline ASR22 Jun
  3. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models7 Oct
  4. 2021CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition14 Jun
  5. 2021Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition28 Feb

Other references · 19

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  4. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  5. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  6. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  7. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 97
  8. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  9. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  10. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 69
  11. 2020Efficient Knowledge Distillation for RNN-Transducer Modelscited by 38
  12. 2022Pseudo Label Is Better Than Human Labelcited by 25
7 more
  1. 2020Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Datacited by 16
  2. 2020Understanding Self-Attention of Self-Supervised Audio Transformerscited by 15
  3. 2017Improved training for online end-to-end speech recognition systemscited by 13
  4. 2022A Unified Cascaded Encoder ASR Model for Dynamic Model Sizescited by 11
  5. 2022Learning a Dual-Mode Speech Recognition Model via Self-Pruningcited by 5
  6. 2022Fast-U2++: Fast and Accurate End-to-End Speech Recognition in Joint CTC/Attention Framescited by 4
  7. 2022Similarity and Content-based Phonetic Self Attention for Speech Recognitioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Delayed-KD: Delayed Knowledge Distillation based CTC for Low-Latency Streaming ASRcited by 1
  2. 2026Alignment-Path Distillation from Non-streaming ASR-LLMs for Streaming Speech Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Joint Optimization of Streaming and Non-Streaming Automatic Speech Recognition with Multi-Decoder and Knowledge Distillation
  2. 2023Reducing the gap between streaming and non-streaming Transducer-based ASR by adaptive two-stage knowledge distillation
  3. 2023DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Model
  4. 2023Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition
  5. 2024Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data
  6. 2021Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models
  7. 2026SENS-ASR: Semantic Embedding injection in Neural-transducer for Streaming Automatic Speech Recognition
  8. 2024Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper