# Exploiting Cross-domain And Cross-Lingual Ultrasound Tongue Imaging Features For Elderly And Dysarthric Speech Recognition

arXiv: https://arxiv.org/abs/2206.07327 · HTML: https://mlsanity.com/paper/2206.07327/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 63 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 15 of 15)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation](https://mlsanity.com/paper/1601.02828/) (2016)
- [Investigation of Data Augmentation Techniques for Disordered Speech Recognition](https://mlsanity.com/paper/2201.05562/) (2022)
- [Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2202.10290/) (2022)
- [Recent Progress in the CUHK Dysarthric Speech Recognition System](https://mlsanity.com/paper/2201.05845/) (2022)
- [UltraSuite: A Repository of Ultrasound and Acoustic Data from Child Speech Therapy Sessions](https://mlsanity.com/paper/1907.00835/) (2019)
- [Spectro-Temporal Deep Features for Disordered Speech Assessment and Recognition](https://mlsanity.com/paper/2201.05554/) (2022)
- [TaL: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos](https://mlsanity.com/paper/2011.09804/) (2020)
- [Exploiting Cross Domain Acoustic-to-articulatory Inverted Features For Disordered Speech Recognition](https://mlsanity.com/paper/2203.10274/) (2022)
- [Updating the silent speech challenge benchmark with deep learning](https://mlsanity.com/paper/1709.06818/) (2017)
- [Speaker-independent classification of phonetic segments from raw ultrasound in child speech](https://mlsanity.com/paper/1907.01413/) (2019)
- [DNN-based Acoustic-to-Articulatory Inversion using Ultrasound Tongue Imaging](https://mlsanity.com/paper/1904.06083/) (2019)
- [Articulatory and bottleneck features for speaker-independent ASR of dysarthric speech](https://mlsanity.com/paper/1905.06533/) (2019)
- [Ultrasound tongue imaging for diarization and alignment of child speech therapy sessions](https://mlsanity.com/paper/1907.00818/) (2019)

## Built on by (most cited first, 7 of 7)

- [Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2302.14564/) (2023)
- [Self-supervised ASR Models and Features For Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2407.13782/) (2024)
- [Use of Speech Impairment Severity for Dysarthric Speech Recognition](https://mlsanity.com/paper/2305.10659/) (2023)
- [A New Benchmark of Aphasia Speech Recognition and Detection Based on E-Branchformer and Multi-task Learning](https://mlsanity.com/paper/2305.13331/) (2023)
- [Homogeneous Speaker Features for On-the-Fly Dysarthric and Elderly Speaker Adaptation](https://mlsanity.com/paper/2407.06310/) (2024)
- [Arabic Dysarthric Speech Recognition Using Adversarial and Signal-Based Augmentation](https://mlsanity.com/paper/2306.04368/) (2023)
- [On-the-fly Routing for Zero-shot MoE Speaker Adaptation of Speech Foundation Models for Dysarthric Speech Recognition](https://mlsanity.com/paper/2505.22072/) (2025)

## Abstract (arXiv)

Articulatory features are inherently invariant to acoustic signal distortion and have been successfully incorporated into automatic speech recognition (ASR) systems designed for normal speech. Their practical application to atypical task domains such as elderly and disordered speech across languages is often limited by the difficulty in collecting such specialist data from target speakers. This paper presents a cross-domain and cross-lingual A2A inversion approach that utilizes the parallel audio and ultrasound tongue imaging (UTI) data of the 24-hour TaL corpus in A2A model pre-training before being cross-domain and cross-lingual adapted to three datasets across two languages: the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech corpora; and the English TORGO dysarthric speech data, to produce UTI based articulatory features. Experiments conducted on three tasks suggested incorporating the generated articulatory features consistently outperformed the baseline TDNN and Conformer ASR systems constructed using acoustic features only by statistically significant word or character error rate reductions up to 4.75%, 2.59% and 2.07% absolute (14.69%, 10.64% and 22.72% relative) after data augmentation, speaker adaptation and cross system multi-pass decoding were applied.