# Adaptation of Tongue Ultrasound-Based Silent Speech Interfaces Using Spatial Transformer Networks

arXiv: https://arxiv.org/abs/2305.19130 · HTML: https://mlsanity.com/paper/2305.19130/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 25 of 25)

- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Vid2speech: Speech Reconstruction from Silent Video](https://mlsanity.com/paper/1701.00495/) (2017)
- [Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Video](https://mlsanity.com/paper/2201.10439/) (2022)
- [End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks](https://mlsanity.com/paper/2104.13332/) (2021)
- [Silent Speech Interfaces for Speech Restoration: A Review](https://mlsanity.com/paper/2009.02110/) (2020)
- [3D Morphable Models as Spatial Transformer Networks](https://mlsanity.com/paper/1708.07199/) (2017)
- [TaL: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos](https://mlsanity.com/paper/2011.09804/) (2020)
- [Image-level Harmonization of Multi-Site Data using Image-and-Spatial Transformer Networks](https://mlsanity.com/paper/2006.16741/) (2020)
- [Reconstructing Speech from Real-Time Articulatory MRI Using Neural Vocoders](https://mlsanity.com/paper/2104.11598/) (2021)
- [Speaker disentanglement in video-to-speech conversion](https://mlsanity.com/paper/2105.09652/) (2021)
- [VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion](https://mlsanity.com/paper/2202.09081/) (2022)
- [SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks](https://mlsanity.com/paper/2303.01758/) (2023)
- [Predicting tongue motion in unlabeled ultrasound videos using convolutional LSTM neural network](https://mlsanity.com/paper/1902.06927/) (2019)
- [Ultrasound-based Articulatory-to-Acoustic Mapping with WaveGlow Speech Synthesis](https://mlsanity.com/paper/2008.03152/) (2020)
- [3D Convolutional Neural Networks for Ultrasound-Based Silent Speech Interfaces](https://mlsanity.com/paper/2104.11532/) (2021)
- [Neural Speaker Embeddings for Ultrasound-based Silent Speech Interfaces](https://mlsanity.com/paper/2106.04552/) (2021)
- [Weakly-supervised Caricature Face Parsing through Domain Adaptation](https://mlsanity.com/paper/1905.05091/) (2019)
- [Ultrasound-based Silent Speech Interface Built on a Continuous Vocoder](https://mlsanity.com/paper/1906.09885/) (2019)
- [Quantification of Transducer Misalignment in Ultrasound Tongue Imaging](https://mlsanity.com/paper/2008.02470/) (2020)
- [Adaptation of Tacotron2-based Text-To-Speech for Articulatory-to-Acoustic Mapping using Ultrasound Tongue Imaging](https://mlsanity.com/paper/2107.12051/) (2021)
- [Ultra2Speech -- A Deep Learning Framework for Formant Frequency Estimation and Tracking from Ultrasound Tongue Images](https://mlsanity.com/paper/2006.16367/) (2020)

## Built on by (most cited first, 1 of 1)

- [Multimodal Deep Learning for Subtype Classification in Breast Cancer Using Histopathological Images and Gene Expression Data](https://mlsanity.com/paper/2503.02849/) (2025)

## Abstract (arXiv)

Thanks to the latest deep learning algorithms, silent speech interfaces (SSI) are now able to synthesize intelligible speech from articulatory movement data under certain conditions. However, the resulting models are rather speaker-specific, making a quick switch between users troublesome. Even for the same speaker, these models perform poorly cross-session, i.e. after dismounting and re-mounting the recording equipment. To aid quick speaker and session adaptation of ultrasound tongue imaging-based SSI models, we extend our deep networks with a spatial transformer network (STN) module, capable of performing an affine transformation on the input images. Although the STN part takes up only about 10% of the network, our experiments show that adapting just the STN module might allow to reduce MSE by 88% on the average, compared to retraining the whole network. The improvement is even larger (around 92%) when adapting the network to different recording sessions from the same speaker.