# Extending Text-to-Speech Synthesis with Articulatory Movement Prediction using Ultrasound Tongue Imaging

arXiv: https://arxiv.org/abs/2107.05550 · HTML: https://mlsanity.com/paper/2107.05550/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models.
- Builds on 9 of its 30 references that are in the map.
- First posted on arXiv 12 Jul 2021.

## Placed in

- Speech and audio language models / [Vocoders and speech synthesis](https://mlsanity.com/speech/vocoders-and-speech-synthesis/)

## Builds on (most cited first, 9 of 9)

- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [TaL: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos](https://mlsanity.com/paper/2011.09804/) (2020)
- [Ultrasound-based Articulatory-to-Acoustic Mapping with WaveGlow Speech Synthesis](https://mlsanity.com/paper/2008.03152/) (2020)
- [DNN-based Acoustic-to-Articulatory Inversion using Ultrasound Tongue Imaging](https://mlsanity.com/paper/1904.06083/) (2019)
- [Quantification of Transducer Misalignment in Ultrasound Tongue Imaging](https://mlsanity.com/paper/2008.02470/) (2020)
- [Synthesis of Tongue Motion and Acoustics from Text using a Multimodal Articulatory Database](https://mlsanity.com/paper/1612.09352/) (2016)

## Abstract (arXiv)

In this paper, we present our first experiments in text-to-articulation prediction, using ultrasound tongue image targets. We extend a traditional (vocoder-based) DNN-TTS framework with predicting PCA-compressed ultrasound images, of which the continuous tongue motion can be reconstructed in synchrony with synthesized speech. We use the data of eight speakers, train fully connected and recurrent neural networks, and show that FC-DNNs are more suitable for the prediction of sequential data than LSTMs, in case of limited training data. Objective experiments and visualized predictions show that the proposed solution is feasible and the generated ultrasound videos are close to natural tongue movement. Articulatory movement prediction from text input can be useful for audiovisual speech synthesis or computer-assisted pronunciation training.