mlsanityData to 19 Oct 2026

Speech Synthesis from Text and Ultrasound Tongue Image-based Articulatory Input

arXiv 2107.02003 ↗First posted 5 Jul 2021eess.AS cs.SDcited by 1

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Articulatory information has been shown to be effective in improving the performance of HMM-based and DNN-based text-to-speech synthesis. Speech synthesis research focuses traditionally on text-to-speech conversion, when the input is text or an estimated linguistic representation, and the target is synthesized speech. However, a research field that has risen in the last decade is articulation-to-speech synthesis (with a target application of a Silent Speech Interface, SSI), when the goal is to synthesize speech from some representation of the movement of the articulatory organs. In this paper, we extend traditional (vocoder-based) DNN-TTS with articulatory input, estimated from ultrasound tongue images. We compare text-only, ultrasound-only, and combined inputs. Using data from eight speakers, we show that that the combined text and articulatory input can have advantages in limited-data scenarios, namely, it may increase the naturalness of synthesized speech compared to single text input. Besides, we analyze the ultrasound tongue recordings of several speakers, and show that misalignments in the ultrasound transducer positioning can have a negative effect on the final synthesis performance.

Key facts. Cited by 1 paper in the map (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 5 of its 31 references that are in the map. First posted on arXiv 5 Jul 2021.

Builds on

5 of 31 references.

References · 5

  1. 2020Silent Speech Interfaces for Speech Restoration: A Reviewcited by 21
  2. 2020TaL: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videoscited by 19
  3. 2020Ultrasound-based Articulatory-to-Acoustic Mapping with WaveGlow Speech Synthesiscited by 10
  4. 2019Ultrasound-based Silent Speech Interface Built on a Continuous Vocodercited by 4
  5. 2020Quantification of Transducer Misalignment in Ultrasound Tongue Imagingcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022vTTS: visual-text to speechcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2021Extending Text-to-Speech Synthesis with Articulatory Movement Prediction using Ultrasound Tongue Imaging
  2. 2021Adaptation of Tacotron2-based Text-To-Speech for Articulatory-to-Acoustic Mapping using Ultrasound Tongue Imaging
  3. 2023Speech Reconstruction from Silent Tongue and Lip Articulation By Pseudo Target Generation and Domain Adversarial Training
  4. 2022Deep Speech Synthesis from Articulatory Representations
  5. 2026Evaluating Speech Articulation Synthesis with Articulatory Phoneme Recognition
  6. 2023Deep Speech Synthesis from MRI-Based Articulatory Representations
  7. 2024An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data
  8. 2024MRI2Speech: Speech Synthesis from Articulatory Movements Recorded by Real-time MRI