mlsanityData to 7 Oct 2026

ILASR: Privacy-Preserving Incremental Learning for Automatic Speech Recognition at Production Scale

arXiv 2207.09078 ↗First posted 19 Jul 2022cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Incremental learning is one paradigm to enable model building and updating at scale with streaming data. For end-to-end automatic speech recognition (ASR) tasks, the absence of human annotated labels along with the need for privacy preserving policies for model building makes it a daunting challenge. Motivated by these challenges, in this paper we use a cloud based framework for production systems to demonstrate insights from privacy preserving incremental learning for automatic speech recognition (ILASR). By privacy preserving, we mean, usage of ephemeral data which are not human annotated. This system is a step forward for production levelASR models for incremental/continual learning that offers near real-time test-bed for experimentation in the cloud for end-to-end ASR, while adhering to privacy-preserving policies. We show that the proposed system can improve the production models significantly(3%) over a new time period of six months even in the absence of human annotated labels with varying levels of weak supervision and large batch sizes in incremental learning. This improvement is 20% over test sets with new words and phrases in the new time period. We demonstrate the effectiveness of model building in a privacy-preserving incremental fashion for ASR while further exploring the utility of having an effective teacher model and use of large batch sizes.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 42 of its 68 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

42 of 68 references.

References · 42

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2020Adaptive Federated Optimizationcited by 1,036
  10. 2019Deep Leakage from Gradientscited by 1,004
  11. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  12. 2019Towards Federated Learning at Scale: System Designcited by 826
30 more
  1. 2019Large Scale Incremental Learningcited by 727
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  3. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  4. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  5. 2018End-to-End Incremental Learningcited by 596
  6. 2020Federated Learning with Matched Averagingcited by 564
  7. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
  8. 2020iDLG: Improved Deep Leakage from Gradientscited by 367
  9. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  10. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  11. 2018An Empirical Model of Large-Batch Trainingcited by 268
  12. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  13. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  14. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  15. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  16. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 100
  17. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  18. 2018Privacy Preserving Machine Learning: Threats and Solutionscited by 78
  19. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  20. 2020Training Speech Recognition Models with Federated Learning: A Quality/Cost Frameworkcited by 46
  21. 2021End-to-End Speech Recognition from Federated Acoustic Modelscited by 30
  22. 2020Improving on-device speaker verification using federated learning with privacycited by 29
  23. 2019Lessons from Building Acoustic Models with a Million Hours of Speechcited by 27
  24. 2018On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descentcited by 25
  25. 2021Federated Acoustic Modeling For Automatic Speech Recognitioncited by 20
  26. 2020Semi-Supervised Learning with Data Augmentation for End-to-End ASRcited by 19
  27. 2019Federated Generative Privacycited by 16
  28. 2019Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutescited by 13
  29. 2021Contrastive Semi-supervised Learning for ASRcited by 11
  30. 2019Exploiting semi-supervised training through a dropout regularization in end-to-end speech recognitioncited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 2.

4 papers.

  1. 2022A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairnesscited by 13
  2. 2024Continuously Learning New Words in Automatic Speech Recognitioncited by 3
  3. 2023Federated Representation Learning for Automatic Speech Recognitioncited by 0
  4. 2024An Efficient Self-Learning Framework For Interactive Spoken Dialog Systemscited by 0