# SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks

arXiv: https://arxiv.org/abs/2303.01758 · HTML: https://mlsanity.com/paper/2303.01758/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 59 references that are in the map.
- First posted on arXiv 3 Mar 2023.

## Builds on (most cited first, 5 of 5)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Improved Speech Reconstruction from Silent Video](https://mlsanity.com/paper/1708.01204/) (2017)
- [Lipreading with Long Short-Term Memory](https://mlsanity.com/paper/1601.08188/) (2016)
- [Updating the silent speech challenge benchmark with deep learning](https://mlsanity.com/paper/1709.06818/) (2017)

## Built on by (most cited first, 5 of 5)

- [A Cross-Modal Approach to Silent Speech with LLM-Enhanced Recognition](https://mlsanity.com/paper/2403.05583/) (2024)
- [Adaptation of Tongue Ultrasound-Based Silent Speech Interfaces Using Spatial Transformer Networks](https://mlsanity.com/paper/2305.19130/) (2023)
- [Transfer Learning for Covert Speech Classification Using EEG Hilbert Envelope and Temporal Fine Structure](https://mlsanity.com/paper/2502.04132/) (2025)
- [NasoVoce: A Nose-Mounted Low-Audibility Speech Interface for Always-Available Speech Interaction](https://mlsanity.com/paper/2603.10324/) (2026)
- [Soft Active Electromyography Interface for Machine Learning-Enabled Silent Speech Recognition](https://mlsanity.com/paper/2608.27048/) (2026)

## Abstract (arXiv)

The availability of digital devices operated by voice is expanding rapidly. However, the applications of voice interfaces are still restricted. For example, speaking in public places becomes an annoyance to the surrounding people, and secret information should not be uttered. Environmental noise may reduce the accuracy of speech recognition. To address these limitations, a system to detect a user's unvoiced utterance is proposed. From internal information observed by an ultrasonic imaging sensor attached to the underside of the jaw, our proposed system recognizes the utterance contents without the user's uttering voice. Our proposed deep neural network model is used to obtain acoustic features from a sequence of ultrasound images. We confirmed that audio signals generated by our system can control the existing smart speakers. We also observed that a user can adjust their oral movement to learn and improve the accuracy of their voice recognition.