# Speech Reconstruction from Silent Tongue and Lip Articulation By Pseudo Target Generation and Domain Adversarial Training

arXiv: https://arxiv.org/abs/2304.05574 · HTML: https://mlsanity.com/paper/2304.05574/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2023.

## Built on by (most cited first, 2 of 2)

- [An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data](https://mlsanity.com/paper/2403.05820/) (2024)
- [Disentangling Granularity: An Implicit Inductive Bias in Factorized VAEs](https://mlsanity.com/paper/2505.24684/) (2025)

## Abstract (arXiv)

This paper studies the task of speech reconstruction from ultrasound tongue images and optical lip videos recorded in a silent speaking mode, where people only activate their intra-oral and extra-oral articulators without producing sound. This task falls under the umbrella of articulatory-to-acoustic conversion, and may also be refered to as a silent speech interface. We propose to employ a method built on pseudo target generation and domain adversarial training with an iterative training strategy to improve the intelligibility and naturalness of the speech recovered from silent tongue and lip articulation. Experiments show that our proposed method significantly improves the intelligibility and naturalness of the reconstructed speech in silent speaking mode compared to the baseline TaLNet model. When using an automatic speech recognition (ASR) model to measure intelligibility, the word error rate (WER) of our proposed method decreases by over 15% compared to the baseline. In addition, our proposed method also outperforms the baseline on the intelligibility of the speech reconstructed in vocalized articulating mode, reducing the WER by approximately 10%.