# Guiding CTC Posterior Spike Timings for Improved Posterior Fusion and Knowledge Distillation

arXiv: https://arxiv.org/abs/1904.08311 · HTML: https://mlsanity.com/paper/1904.08311/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2019.

## Built on by (most cited first, 18 of 18)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition](https://mlsanity.com/paper/2103.00422/) (2021)
- [Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence tasks](https://mlsanity.com/paper/2210.07499/) (2022)
- [CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition](https://mlsanity.com/paper/2106.07734/) (2021)
- [Integrating Dialog History into End-to-End Spoken Language Understanding Systems](https://mlsanity.com/paper/2108.08405/) (2021)
- [RNN Transducer Models For Spoken Language Understanding](https://mlsanity.com/paper/2104.03842/) (2021)
- [Distilling the Knowledge of BERT for CTC-based ASR](https://mlsanity.com/paper/2209.02030/) (2022)
- [Exploiting Large-scale Teacher-Student Training for On-device Acoustic Models](https://mlsanity.com/paper/2106.06126/) (2021)
- [Inter-KD: Intermediate Knowledge Distillation for CTC-Based Automatic Speech Recognition](https://mlsanity.com/paper/2211.15075/) (2022)
- [Guiding Frame-Level CTC Alignments Using Self-knowledge Distillation](https://mlsanity.com/paper/2406.07909/) (2024)
- [HMM-Free Encoder Pre-Training for Streaming RNN Transducer](https://mlsanity.com/paper/2104.10764/) (2021)
- [EM-Network: Oracle Guided Self-distillation for Sequence Learning](https://mlsanity.com/paper/2306.10058/) (2023)
- [Delayed-KD: Delayed Knowledge Distillation based CTC for Low-Latency Streaming ASR](https://mlsanity.com/paper/2505.22069/) (2025)
- [Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Models](https://mlsanity.com/paper/2111.03664/) (2021)
- [Improving End-to-End Models for Set Prediction in Spoken Language Understanding](https://mlsanity.com/paper/2201.12105/) (2022)
- [Alternating Weak Triphone/BPE Alignment Supervision from Hybrid Model Improves End-to-End ASR](https://mlsanity.com/paper/2402.15594/) (2024)
- [A Differentiable Alignment Framework for Sequence-to-Sequence Modeling via Optimal Transport](https://mlsanity.com/paper/2502.01588/) (2025)
- [Analyzing the Importance of Blank for CTC-Based Knowledge Distillation](https://mlsanity.com/paper/2506.01503/) (2025)

## Abstract (arXiv)

Conventional automatic speech recognition (ASR) systems trained from frame-level alignments can easily leverage posterior fusion to improve ASR accuracy and build a better single model with knowledge distillation. End-to-end ASR systems trained using the Connectionist Temporal Classification (CTC) loss do not require frame-level alignment and hence simplify model training. However, sparse and arbitrary posterior spike timings from CTC models pose a new set of challenges in posterior fusion from multiple models and knowledge distillation between CTC models. We propose a method to train a CTC model so that its spike timings are guided to align with those of a pre-trained guiding CTC model. As a result, all models that share the same guiding model have aligned spike timings. We show the advantage of our method in various scenarios including posterior fusion of CTC models and knowledge distillation between CTC models with different architectures. With the 300-hour Switchboard training data, the single word CTC model distilled from multiple models improved the word error rates to 13.7%/23.1% from 14.9%/24.1% on the Hub5 2000 Switchboard/CallHome test sets without using any data augmentation, language model, or complex decoder.