# Delayed-KD: Delayed Knowledge Distillation based CTC for Low-Latency Streaming ASR

arXiv: https://arxiv.org/abs/2505.22069 · HTML: https://mlsanity.com/paper/2505.22069/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 23 of 23)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition](https://mlsanity.com/paper/2110.03370/) (2021)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2203.15455/) (2022)
- [The PyTorch-Kaldi Speech Recognition Toolkit](https://mlsanity.com/paper/1811.07453/) (2018)
- [U2++: Unified Two-pass Bidirectional End-to-end Model for Speech Recognition](https://mlsanity.com/paper/2106.05642/) (2021)
- [Dissecting User-Perceived Latency of On-Device E2E Speech Recognition](https://mlsanity.com/paper/2104.02207/) (2021)
- [Guiding CTC Posterior Spike Timings for Improved Posterior Fusion and Knowledge Distillation](https://mlsanity.com/paper/1904.08311/) (2019)
- [Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition](https://mlsanity.com/paper/2103.00422/) (2021)
- [Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence tasks](https://mlsanity.com/paper/2210.07499/) (2022)
- [Mamba for Streaming ASR Combined with Unimodal Aggregation](https://mlsanity.com/paper/2410.00070/) (2024)
- [Delay-penalized transducer for low-latency streaming ASR](https://mlsanity.com/paper/2211.00490/) (2022)
- [Peak-First CTC: Reducing the Peak Latency of CTC Models by Applying Peak-First Regularization](https://mlsanity.com/paper/2211.03284/) (2022)
- [XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Models](https://mlsanity.com/paper/2407.04439/) (2024)
- [Fast-U2++: Fast and Accurate End-to-End Speech Recognition in Joint CTC/Attention Frames](https://mlsanity.com/paper/2211.00941/) (2022)
- [Reducing the gap between streaming and non-streaming Transducer-based ASR by adaptive two-stage knowledge distillation](https://mlsanity.com/paper/2306.15171/) (2023)
- [CUSIDE-T: Chunking, Simulating Future and Decoding for Transducer based Streaming ASR](https://mlsanity.com/paper/2407.10255/) (2024)
- [Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer](https://mlsanity.com/paper/2308.16415/) (2023)
- [Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition](https://mlsanity.com/paper/2312.12783/) (2023)
- [Guiding Frame-Level CTC Alignments Using Self-knowledge Distillation](https://mlsanity.com/paper/2406.07909/) (2024)

## Built on by (most cited first, 1 of 1)

- [Alignment-Path Distillation from Non-streaming ASR-LLMs for Streaming Speech Recognition](https://mlsanity.com/paper/2609.20121/) (2026)

## Abstract (arXiv)

CTC-based streaming ASR has gained significant attention in real-world applications but faces two main challenges: accuracy degradation in small chunks and token emission latency. To mitigate these challenges, we propose Delayed-KD, which applies delayed knowledge distillation on CTC posterior probabilities from a non-streaming to a streaming model. Specifically, with a tiny chunk size, we introduce a Temporal Alignment Buffer (TAB) that defines a relative delay range compared to the non-streaming teacher model to align CTC outputs and mitigate non-blank token mismatches. Additionally, TAB enables fine-grained control over token emission delay. Experiments on 178-hour AISHELL-1 and 10,000-hour WenetSpeech Mandarin datasets show consistent superiority of Delayed-KD. Impressively, Delayed-KD at 40 ms latency achieves a lower character error rate (CER) of 5.42% on AISHELL-1, comparable to the competitive U2++ model running at 320 ms latency.