Training Large ASR Encoders with Differential Privacy
Not on the map. Why?
Abstract (arXiv)
Self-supervised learning (SSL) methods for large speech models have proven to be highly effective at ASR. With the interest in public deployment of large pre-trained models, there is a rising concern for unintended memorization and leakage of sensitive data points from the training data. In this paper, we apply differentially private (DP) pre-training to a SOTA Conformer-based encoder, and study its performance on a downstream ASR task assuming the fine-tuning data is public. This paper is the first to apply DP to SSL for ASR, investigating the DP noise tolerance of the BEST-RQ pre-training method. Notably, we introduce a novel variant of model pruning called gradient-based layer freezing that provides strong improvements in privacy-utility-compute trade-offs. Our approach yields a LibriSpeech test-clean/other WER (%) of 3.78/ 8.41 with (10, 1e^-9)-DP for extrapolation towards low dataset scales, and 2.81/ 5.89 with (10, 7.9e^-11)-DP for extrapolation towards high scales.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 21 Sep 2024.
Builds on
31 of 44 references.
References · 31
- 2017Attention Is All You Needcited by 60,361
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2016Deep Learning with Differential Privacycited by 2,728
- 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
- 2020Extracting Training Data from Large Language Modelscited by 2,189
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
- 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
- 2023Extracting Training Data from Diffusion Modelscited by 732
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
- 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
- 2021Differentially Private Fine-tuning of Language Modelscited by 400
- 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
19 more
- 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
- 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 204
- 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
- 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
- 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
- 2021Large-Scale Differentially Private BERTcited by 118
- 2022Measuring Forgetting of Memorized Training Examplescited by 118
- 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
- 2019Jasper: An End-to-End Convolutional Neural Acoustic Modelcited by 76
- 2020Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorizationcited by 58
- 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
- 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
- 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 48
- 2023Why Is Public Pretraining Necessary for Private Model Training?cited by 37
- 2022Extracting Targeted Training Data from ASR Models, and How to Mitigate Itcited by 7
- 2023Unintended Memorization in Large ASR Models, and How to Mitigate Itcited by 5
- 2023Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers and Gradient Clippingcited by 4
- 2024Noise Masking Attacks and Defenses for Pretrained Speech Modelscited by 3
- 2023Inference and Interference: The Role of Clipping, Pruning and Loss Landscapes in Differentially Private Stochastic Gradient Descentcited by 1
Built on by
Citing papers per year: 2025: 2.
2 papers.