mlsanityData to 7 Oct 2026

Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR

arXiv 2210.05793 ↗First posted 11 Oct 2022cs.LG cs.CL cs.SDcited by 10

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation is an effective machine learning technique to transfer knowledge from a teacher model to a smaller student model, especially with unlabeled data. In this paper, we focus on knowledge distillation for the RNN-T model, which is widely used in state-of-the-art (SoTA) automatic speech recognition (ASR). Specifically, we compared using soft and hard target distillation to train large-scaleRNN-T models on the LibriSpeech/LibriLight public dataset (60k hours) and our in-house data (600k hours). We found that hard tar-gets are more effective when the teacher and student have different architecture, such as large teacher and small streaming student. On the other hand, soft target distillation works better in self-training scenario like iterative large teacher training. For a large model with0.6B weights, we achieve a new SoTA word error rate (WER) on LibriSpeech (8% relative improvement on dev-other) using Noisy Student Training with soft target distillation. It also allows our production teacher to adapt new data domain continuously.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

29 of 39 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2020Knowledge Distillation: A Surveycited by 1,578
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  9. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  10. 2019When Does Label Smoothing Help?cited by 985
  11. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  12. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
17 more
  1. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  2. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  3. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  4. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  5. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  6. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
  7. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 98
  8. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  9. 2019Recognizing long-form speech using streaming end-to-end modelscited by 77
  10. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 68
  11. 2018Toward domain-invariant speech recognition via large scale trainingcited by 35
  12. 2018Cycle-consistency training for end-to-end speech recognitioncited by 31
  13. 2020Efficient Knowledge Distillation for RNN-Transducer Modelscited by 29
  14. 2022Pseudo Label Is Better Than Human Labelcited by 25
  15. 2021Injecting Text in Self-Supervised Speech Pretrainingcited by 24
  16. 2020Why distillation helps: a statistical perspectivecited by 21
  17. 2022Solving ImageNet: a Unified Scheme for Training any Backbone to Top Resultscited by 8

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 4; 2025: 2.

10 papers.

  1. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Datacited by 9
  2. 2022Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems A case study for Modern Greekcited by 7
  3. 2024Generation, Distillation and Evaluation of Motivational Interviewing-Style Reflections with a Foundational Language Modelcited by 5
  4. 2023Robust Knowledge Distillation from RNN-T Models With Noisy Training Labels Using Full-Sum Losscited by 2
  5. 2023Revisiting the Entropy Semiring for Neural Speech Recognitioncited by 1
  6. 2024Optimizing Vision Transformers with Data-Free Knowledge Transfercited by 1
  7. 2024DETECLAP: Enhancing Audio-Visual Representation Learning with Object Informationcited by 1
  8. 2023O-1: Self-training with Oracle and 1-best Hypothesiscited by 0
  9. 2025KDC-Diff: A Latent-Aware Diffusion Model with Knowledge Retention for Memory-Efficient Image Generationcited by 0
  10. 2025Analyzing the Importance of Blank for CTC-Based Knowledge Distillationcited by 0