# Knowledge Transfer and Distillation from Autoregressive to Non-Autoregressive Speech Recognition

arXiv: https://arxiv.org/abs/2207.10600 · HTML: https://mlsanity.com/paper/2207.10600/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2022.

## Built on by (most cited first, 2 of 2)

- [Data-Efficient On-Policy Distillation for Automatic Speech Recognition](https://mlsanity.com/paper/2605.28139/) (2026)
- [NADIR: Differential Attention Flow for Non-Autoregressive Transliteration in Indic Languages](https://mlsanity.com/paper/2601.12389/) (2026)

## Abstract (arXiv)

Modern non-autoregressive~(NAR) speech recognition systems aim to accelerate the inference speed; however, they suffer from performance degradation compared with autoregressive~(AR) models as well as the huge model size issue. We propose a novel knowledge transfer and distillation architecture that leverages knowledge from AR models to improve the NAR performance while reducing the model's size. Frame- and sequence-level objectives are well-designed for transfer learning. To further boost the performance of NAR, a beam search method on Mask-CTC is developed to enlarge the search space during the inference stage. Experiments show that the proposed NAR beam search relatively reduces CER by over 5% on AISHELL-1 benchmark with a tolerable real-time-factor~(RTF) increment. By knowledge transfer, the NAR student who has the same size as the AR teacher obtains relative CER reductions of 8/16% on AISHELL-1 dev/test sets, and over 25% relative WER reductions on LibriSpeech test-clean/other sets. Moreover, the ~9x smaller NAR models achieve ~25% relative CER/WER reductions on both AISHELL-1 and LibriSpeech benchmarks with the proposed knowledge transfer and distillation.