mlsanityData to 7 Oct 2026

Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper

arXiv 2409.13499 ↗First posted 20 Sep 2024cs.CL cs.SD eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

The training of automatic speech recognition (ASR) with little to no supervised data remains an open question. In this work, we demonstrate that streaming Transformer-Transducer (TT) models can be trained from scratch in consumer and accessible GPUs in their entirety with pseudo-labeled (PL) speech from foundational speech models (FSM). This allows training a robust ASR model just in one stage and does not require large data and computational budget compared to the two-step scenario with pre-training and fine-tuning. We perform a comprehensive ablation on different aspects of PL-based streaming TT models such as the impact of (1) shallow fusion of n-gram LMs, (2) contextual biasing with named entities, (3) chunk-wise decoding for low-latency streaming applications, and (4) TT overall performance as the function of the FSM size. Our results demonstrate that TT can be trained from scratch without supervised data, even with very noisy PLs. We validate the proposed framework on 6 languages from CommonVoice and propose multiple heuristics to filter out hallucinated PLs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 46 of its 75 references that are in the map. First posted on arXiv 20 Sep 2024.

Builds on

46 of 75 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  9. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  10. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  11. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  12. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 320
34 more
  1. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  2. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  3. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  4. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  5. 2023End-to-End Speech Recognition: A Surveycited by 127
  6. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 120
  7. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 109
  8. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
  9. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  10. 2019Transformer-Transducer: End-to-End Speech Recognition with Self-Attentioncited by 68
  11. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 68
  12. 2023TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudiocited by 56
  13. 2020SlimIPL: Language-Model-Free Iterative Pseudo-Labelingcited by 54
  14. 2020Towards Fast and Accurate Streaming End-to-End ASRcited by 47
  15. 2022Pruned RNN-T for fast, memory-efficient ASR trainingcited by 46
  16. 2017Exploring Neural Transducers for End-to-End Speech Recognitioncited by 43
  17. 2021Momentum Pseudo-Labeling for Semi-Supervised Speech Recognitioncited by 37
  18. 2021Lhotse: a speech data representation library for the modern deep learning ecosystemcited by 33
  19. 2022How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communicationscited by 30
  20. 2023Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentationcited by 30
  21. 2020Efficient Knowledge Distillation for RNN-Transducer Modelscited by 29
  22. 2018End-to-end contextual speech recognition using class language models and a token passing decodercited by 26
  23. 2021Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalancited by 26
  24. 2022Pseudo Label Is Better Than Human Labelcited by 25
  25. 2021Pseudo-Labeling for Massively Multilingual Speech Recognitioncited by 23
  26. 2023Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Expertscited by 15
  27. 2022Variable Attention Masking for Configurable Transformer Transducer Speech Recognitioncited by 14
  28. 2021Contextual Semi-Supervised Learning: An Approach To Leverage Air-Surveillance and Untranscribed ATC Data in ASR Systemscited by 12
  29. 2021Omni-sparsity DNN: Fast Sparsity Optimization for On-Device Streaming E2E ASR via Supernetcited by 12
  30. 2022Continuous Pseudo-Labeling from the Startcited by 11
  31. 2021Spell my name: keyword boosted speech recognitioncited by 10
  32. 2023Bypass Temporal Classification: Weakly Supervised Automatic Speech Recognition with Imperfect Transcriptscited by 10
  33. 2022Continuous Soft Pseudo-Labeling in ASRcited by 6
  34. 2024XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026StreamHear: Domain-Adapted Pseudo-Labeling for Semi-Supervised Streaming Speech Recognitioncited by 0