# An Experimental Study on Private Aggregation of Teacher Ensemble Learning for End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/2210.05614 · HTML: https://mlsanity.com/paper/2210.05614/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 45 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [Scalable Private Learning with PATE](https://mlsanity.com/paper/1802.08908/) (2018)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Federated Learning for Keyword Spotting](https://mlsanity.com/paper/1810.05512/) (2018)
- [Federated Quantum Machine Learning](https://mlsanity.com/paper/2103.12010/) (2021)
- [Decentralizing Feature Extraction with Quantum Convolutional Neural Network for Automatic Speech Recognition](https://mlsanity.com/paper/2010.13309/) (2020)
- [The VoicePrivacy 2020 Challenge: Results and findings](https://mlsanity.com/paper/2109.00648/) (2021)
- [Federated Heavy Hitters Discovery with Differential Privacy](https://mlsanity.com/paper/1902.08534/) (2019)
- [Personalized Speech recognition on mobile devices](https://mlsanity.com/paper/1603.03185/) (2016)
- [Multi-task Language Modeling for Improving Speech Recognition of Rare Words](https://mlsanity.com/paper/2011.11715/) (2020)
- [Federated Acoustic Modeling For Automatic Speech Recognition](https://mlsanity.com/paper/2102.04429/) (2021)
- [Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learning](https://mlsanity.com/paper/2106.00273/) (2021)
- [PATE-AAE: Incorporating Adversarial Autoencoder into Private Aggregation of Teacher Ensembles for Spoken Command Classification](https://mlsanity.com/paper/2104.01271/) (2021)
- [Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognition](https://mlsanity.com/paper/2005.09310/) (2020)
- [Speech Recovery for Real-World Self-powered Intermittent Devices](https://mlsanity.com/paper/2106.05229/) (2021)
- [Mitigating Closed-model Adversarial Examples with Bayesian Neural Modeling for Enhanced End-to-End Speech Recognition](https://mlsanity.com/paper/2202.08532/) (2022)

## Built on by (most cited first, 1 of 1)

- [An Ensemble Teacher-Student Learning Approach with Poisson Sub-sampling to Differential Privacy Preserving Speech Recognition](https://mlsanity.com/paper/2210.06382/) (2022)

## Abstract (arXiv)

Differential privacy (DP) is one data protection avenue to safeguard user information used for training deep models by imposing noisy distortion on privacy data. Such a noise perturbation often results in a severe performance degradation in automatic speech recognition (ASR) in order to meet a privacy budget varepsilon. Private aggregation of teacher ensemble (PATE) utilizes ensemble probabilities to improve ASR accuracy when dealing with the noise effects controlled by small values of varepsilon. We extend PATE learning to work with dynamic patterns, namely speech utterances, and perform a first experimental demonstration that it prevents acoustic data leakage in ASR training. We evaluate three end-to-end deep models, including LAS, hybrid CTC/attention, and RNN transducer, on the open-source LibriSpeech and TIMIT corpora. PATE learning-enhanced ASR models outperform the benchmark DP-SGD mechanisms, especially under strict DP budgets, giving relative word error rate reductions between 26.2% and 27.5% for an RNN transducer model evaluated with LibriSpeech. We also introduce a DP-preserving ASR solution for pretraining on public speech corpora.