# Speech Synthesis from Text and Ultrasound Tongue Image-based Articulatory Input

arXiv: https://arxiv.org/abs/2107.02003 · HTML: https://mlsanity.com/paper/2107.02003/

## Key facts

- Cited by 1 papers in the map (data through 19 Oct 2026).
- Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models.
- Builds on 5 of its 31 references that are in the map.
- First posted on arXiv 5 Jul 2021.

## Placed in

- Speech and audio language models / [Vocoders and speech synthesis](https://mlsanity.com/speech/vocoders-and-speech-synthesis/)

## Builds on (most cited first, 5 of 5)

- [Silent Speech Interfaces for Speech Restoration: A Review](https://mlsanity.com/paper/2009.02110/) (2020)
- [TaL: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos](https://mlsanity.com/paper/2011.09804/) (2020)
- [Ultrasound-based Articulatory-to-Acoustic Mapping with WaveGlow Speech Synthesis](https://mlsanity.com/paper/2008.03152/) (2020)
- [Ultrasound-based Silent Speech Interface Built on a Continuous Vocoder](https://mlsanity.com/paper/1906.09885/) (2019)
- [Quantification of Transducer Misalignment in Ultrasound Tongue Imaging](https://mlsanity.com/paper/2008.02470/) (2020)

## Built on by (most cited first, 1 of 1)

- [vTTS: visual-text to speech](https://mlsanity.com/paper/2203.14725/) (2022)

## Abstract (arXiv)

Articulatory information has been shown to be effective in improving the performance of HMM-based and DNN-based text-to-speech synthesis. Speech synthesis research focuses traditionally on text-to-speech conversion, when the input is text or an estimated linguistic representation, and the target is synthesized speech. However, a research field that has risen in the last decade is articulation-to-speech synthesis (with a target application of a Silent Speech Interface, SSI), when the goal is to synthesize speech from some representation of the movement of the articulatory organs. In this paper, we extend traditional (vocoder-based) DNN-TTS with articulatory input, estimated from ultrasound tongue images. We compare text-only, ultrasound-only, and combined inputs. Using data from eight speakers, we show that that the combined text and articulatory input can have advantages in limited-data scenarios, namely, it may increase the naturalness of synthesized speech compared to single text input. Besides, we analyze the ultrasound tongue recordings of several speakers, and show that misalignments in the ultrasound transducer positioning can have a negative effect on the final synthesis performance.