mlsanityData to 19 Oct 2026

Extending Text-to-Speech Synthesis with Articulatory Movement Prediction using Ultrasound Tongue Imaging

arXiv 2107.05550 ↗First posted 12 Jul 2021eess.AS cs.SDcited by 0

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

In this paper, we present our first experiments in text-to-articulation prediction, using ultrasound tongue image targets. We extend a traditional (vocoder-based) DNN-TTS framework with predicting PCA-compressed ultrasound images, of which the continuous tongue motion can be reconstructed in synchrony with synthesized speech. We use the data of eight speakers, train fully connected and recurrent neural networks, and show that FC-DNNs are more suitable for the prediction of sequential data than LSTMs, in case of limited training data. Objective experiments and visualized predictions show that the proposed solution is feasible and the generated ultrasound videos are close to natural tongue movement. Articulatory movement prediction from text input can be useful for audiovisual speech synthesis or computer-assisted pronunciation training.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 9 of its 30 references that are in the map. First posted on arXiv 12 Jul 2021.

Builds on

9 of 30 references.

References · 9

  1. 2016WaveNet: A Generative Model for Raw Audiocited by 1,880
  2. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 1,023
  3. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 682
  4. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 365
  5. 2020TaL: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videoscited by 19
  6. 2020Ultrasound-based Articulatory-to-Acoustic Mapping with WaveGlow Speech Synthesiscited by 10
  7. 2019DNN-based Acoustic-to-Articulatory Inversion using Ultrasound Tongue Imagingcited by 9
  8. 2020Quantification of Transducer Misalignment in Ultrasound Tongue Imagingcited by 4
  9. 2016Synthesis of Tongue Motion and Acoustics from Text using a Multimodal Articulatory Databasecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Speech Synthesis from Text and Ultrasound Tongue Image-based Articulatory Input
  2. 2021Adaptation of Tacotron2-based Text-To-Speech for Articulatory-to-Acoustic Mapping using Ultrasound Tongue Imaging
  3. 2024Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech
  4. 2023Speech Reconstruction from Silent Tongue and Lip Articulation By Pseudo Target Generation and Domain Adversarial Training
  5. 2024An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data
  6. 2021Towards a practical lip-to-speech conversion system using deep neural networks and mobile application frontend
  7. 2024Complete reconstruction of the tongue contour through acoustic to articulatory inversion using real-time MRI data
  8. 2025A Speech-to-Video Synthesis Approach Using Spatio-Temporal Diffusion for Vocal Tract MRI