# Arabic Dysarthric Speech Recognition Using Adversarial and Signal-Based Augmentation

arXiv: https://arxiv.org/abs/2306.04368 · HTML: https://mlsanity.com/paper/2306.04368/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [MOSNet: Deep Learning based Objective Assessment for Voice Conversion](https://mlsanity.com/paper/1904.08352/) (2019)
- [Investigation of Data Augmentation Techniques for Disordered Speech Recognition](https://mlsanity.com/paper/2201.05562/) (2022)
- [Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2202.10290/) (2022)
- [Adversarial Data Augmentation for Disordered Speech Recognition](https://mlsanity.com/paper/2108.00899/) (2021)
- [Improved End-to-End Dysarthric Speech Recognition via Meta-learning Based Model Re-initialization](https://mlsanity.com/paper/2011.01686/) (2020)
- [Cross-lingual Self-Supervised Speech Representations for Improved Dysarthric Speech Recognition](https://mlsanity.com/paper/2204.01670/) (2022)
- [EasyCall corpus: a dysarthric speech dataset](https://mlsanity.com/paper/2104.02542/) (2021)
- [Synthesizing Dysarthric Speech Using Multi-talker TTS for Dysarthric Speech Recognition](https://mlsanity.com/paper/2201.11571/) (2022)
- [Variational Auto-Encoder Based Variability Encoding for Dysarthric Speech Recognition](https://mlsanity.com/paper/2201.09422/) (2022)
- [Exploiting Cross-domain And Cross-Lingual Ultrasound Tongue Imaging Features For Elderly And Dysarthric Speech Recognition](https://mlsanity.com/paper/2206.07327/) (2022)

## Abstract (arXiv)

Despite major advancements in Automatic Speech Recognition (ASR), the state-of-the-art ASR systems struggle to deal with impaired speech even with high-resource languages. In Arabic, this challenge gets amplified, with added complexities in collecting data from dysarthric speakers. In this paper, we aim to improve the performance of Arabic dysarthric automatic speech recognition through a multi-stage augmentation approach. To this effect, we first propose a signal-based approach to generate dysarthric Arabic speech from healthy Arabic speech by modifying its speed and tempo. We also propose a second stage Parallel Wave Generative (PWG) adversarial model that is trained on an English dysarthric dataset to capture language-independant dysarthric speech patterns and further augment the signal-adjusted speech samples. Furthermore, we propose a fine-tuning and text-correction strategies for Arabic Conformer at different dysarthric speech severity levels. Our fine-tuned Conformer achieved 18% Word Error Rate (WER) and 17.2% Character Error Rate (CER) on synthetically generated dysarthric speech from the Arabic commonvoice speech dataset. This shows significant WER improvement of 81.8% compared to the baseline model trained solely on healthy data. We perform further validation on real English dysarthric speech showing a WER improvement of 124% compared to the baseline trained only on healthy English LJSpeech dataset.