# End-to-end speech recognition modeling from de-identified data

arXiv: https://arxiv.org/abs/2207.05469 · HTML: https://mlsanity.com/paper/2207.05469/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 12 Jul 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [De-identification of Patient Notes with Recurrent Neural Networks](https://mlsanity.com/paper/1606.03475/) (2016)
- [A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition](https://mlsanity.com/paper/1807.10857/) (2018)
- [Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech](https://mlsanity.com/paper/2005.05106/) (2020)
- [Joint Speech Recognition and Speaker Diarization via Sequence Transduction](https://mlsanity.com/paper/1907.05337/) (2019)
- [On Addressing Practical Challenges for RNN-Transducer](https://mlsanity.com/paper/2105.00858/) (2021)
- [Privacy Guarantees for De-identifying Text Transformations](https://mlsanity.com/paper/2008.03101/) (2020)
- [A Deep Learning Architecture for De-identification of Patient Notes: Implementation and Evaluation](https://mlsanity.com/paper/1810.01570/) (2018)
- [Decoupled Federated Learning for ASR with Non-IID Data](https://mlsanity.com/paper/2206.09102/) (2022)

## Built on by (most cited first, 1 of 1)

- [Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2502.00451/) (2025)

## Abstract (arXiv)

De-identification of data used for automatic speech recognition modeling is a critical component in protecting privacy, especially in the medical domain. However, simply removing all personally identifiable information (PII) from end-to-end model training data leads to a significant performance degradation in particular for the recognition of names, dates, locations, and words from similar categories. We propose and evaluate a two-step method for partially recovering this loss. First, PII is identified, and each occurrence is replaced with a random word sequence of the same category. Then, corresponding audio is produced via text-to-speech or by splicing together matching audio fragments extracted from the corpus. These artificial audio/label pairs, together with speaker turns from the original data without PII, are used to train models. We evaluate the performance of this method on in-house data of medical conversations and observe a recovery of almost the entire performance degradation in the general word error rate while still maintaining a strong diarization performance. Our main focus is the improvement of recall and precision in the recognition of PII-related words. Depending on the PII category, between 50% - 90% of the performance degradation can be recovered using our proposed method.