# Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

arXiv: https://arxiv.org/abs/2205.06445 · HTML: https://mlsanity.com/paper/2205.06445/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 139 references that are in the map.
- First posted on arXiv 13 May 2022.

## Builds on (most cited first, 30 of 40)

- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Alzheimer's Dementia Recognition through Spontaneous Speech: The ADReSS Challenge](https://mlsanity.com/paper/2004.06833/) (2020)
- [Transformer-based Acoustic Modeling for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.09799/) (2019)
- [StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks](https://mlsanity.com/paper/1806.02169/) (2018)
- [CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversion](https://mlsanity.com/paper/1904.04631/) (2019)
- [Personalizing ASR for Dysarthric and Accented Speech with Limited Data](https://mlsanity.com/paper/1907.13511/) (2019)
- [Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation](https://mlsanity.com/paper/1601.02828/) (2016)
- [Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks](https://mlsanity.com/paper/1704.00849/) (2017)
- [Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Data](https://mlsanity.com/paper/2203.15321/) (2022)
- [Back-Translation-Style Data Augmentation for End-to-End ASR](https://mlsanity.com/paper/1807.10893/) (2018)
- [Investigation of Data Augmentation Techniques for Disordered Speech Recognition](https://mlsanity.com/paper/2201.05562/) (2022)
- [Code-switching Sentence Generation by Generative Adversarial Networks and its Application to Data Augmentation](https://mlsanity.com/paper/1811.02356/) (2018)
- [Adversarial Data Augmentation for Disordered Speech Recognition](https://mlsanity.com/paper/2108.00899/) (2021)
- [Improved End-to-End Dysarthric Speech Recognition via Meta-learning Based Model Re-initialization](https://mlsanity.com/paper/2011.01686/) (2020)
- [Recent Progress in the CUHK Dysarthric Speech Recognition System](https://mlsanity.com/paper/2201.05845/) (2022)
- [GAN Vocoder: Multi-Resolution Discriminator Is All You Need](https://mlsanity.com/paper/2103.05236/) (2021)
- [EasyCall corpus: a dysarthric speech dataset](https://mlsanity.com/paper/2104.02542/) (2021)
- [SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translation](https://mlsanity.com/paper/2002.12231/) (2020)
- [Spectro-Temporal Deep Features for Disordered Speech Assessment and Recognition](https://mlsanity.com/paper/2201.05554/) (2022)
- [Towards Identity Preserving Normal to Dysarthric Voice Conversion](https://mlsanity.com/paper/2110.08213/) (2021)
- [Synthesizing Dysarthric Speech Using Multi-talker TTS for Dysarthric Speech Recognition](https://mlsanity.com/paper/2201.11571/) (2022)
- [Many-to-Many Voice Conversion using Conditional Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/2002.06328/) (2020)
- [Improving Code-switching Language Modeling with Artificially Generated Texts using Cycle-consistent Adversarial Networks](https://mlsanity.com/paper/2112.06327/) (2021)

## Built on by (most cited first, 16 of 16)

- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2302.14564/) (2023)
- [Self-supervised ASR Models and Features For Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2407.13782/) (2024)
- [Use of Speech Impairment Severity for Dysarthric Speech Recognition](https://mlsanity.com/paper/2305.10659/) (2023)
- [Accurate synthesis of Dysarthric Speech for ASR data augmentation](https://mlsanity.com/paper/2308.08438/) (2023)
- [DuTa-VC: A Duration-aware Typical-to-atypical Voice Conversion Approach with Diffusion Probabilistic Model](https://mlsanity.com/paper/2306.10588/) (2023)
- [Learning Separable Hidden Unit Contributions for Speaker-Adaptive Lip-Reading](https://mlsanity.com/paper/2310.05058/) (2023)
- [Homogeneous Speaker Features for On-the-Fly Dysarthric and Elderly Speaker Adaptation](https://mlsanity.com/paper/2407.06310/) (2024)
- [Not All Errors Are Equal: Investigation of Speech Recognition Errors in Alzheimer's Disease Detection](https://mlsanity.com/paper/2412.06332/) (2024)
- [DyPCL: Dynamic Phoneme-level Contrastive Learning for Dysarthric Speech Recognition](https://mlsanity.com/paper/2501.19010/) (2025)
- [Regularized Federated Learning for Privacy-Preserving Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2506.11069/) (2025)
- [Fairness in Dysarthric Speech Synthesis: Understanding Intrinsic Bias in Dysarthric Speech Cloning using F5-TTS](https://mlsanity.com/paper/2508.05102/) (2025)
- [Controllable Dysarthric Speech Synthesis with Patient-Specific Conditioning for Speaker-Diverse ASR Augmentation](https://mlsanity.com/paper/2602.08696/) (2026)
- [Elderly-Contextual Data Augmentation via Speech Synthesis for Elderly ASR](https://mlsanity.com/paper/2604.24770/) (2026)
- [Towards Personalized Federated Learning for Dysarthric Speech Recognition](https://mlsanity.com/paper/2606.13253/) (2026)
- [Augmenting Dysarthric Speech Severity Assessment with MOS Supervision](https://mlsanity.com/paper/2606.18645/) (2026)

## Abstract (arXiv)

Despite the rapid progress of automatic speech recognition (ASR) technologies targeting normal speech, accurate recognition of dysarthric and elderly speech remains highly challenging tasks to date. It is difficult to collect large quantities of such data for ASR system development due to the mobility issues often found among these users. To this end, data augmentation techniques play a vital role. In contrast to existing data augmentation techniques only modifying the speaking rate or overall shape of spectral contour, fine-grained spectro-temporal differences between dysarthric, elderly and normal speech are modelled using a novel set of speaker dependent (SD) generative adversarial networks (GAN) based data augmentation approaches in this paper. These flexibly allow both: a) temporal or speed perturbed normal speech spectra to be modified and closer to those of an impaired speaker when parallel speech data is available; and b) for non-parallel data, the SVD decomposed normal speech spectral basis features to be transformed into those of a target elderly speaker before being re-composed with the temporal bases to produce the augmented data for state-of-the-art TDNN and Conformer ASR system training. Experiments are conducted on four tasks: the English UASpeech and TORGO dysarthric speech corpora; the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech datasets. The proposed GAN based data augmentation approaches consistently outperform the baseline speed perturbation method by up to 0.91% and 3.0% absolute (9.61% and 6.4% relative) WER reduction on the TORGO and DementiaBank data respectively. Consistent performance improvements are retained after applying LHUC based speaker adaptation.