# Towards a practical lip-to-speech conversion system using deep neural networks and mobile application frontend

arXiv: https://arxiv.org/abs/2104.14467 · HTML: https://mlsanity.com/paper/2104.14467/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 10 of 10)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Vid2speech: Speech Reconstruction from Silent Video](https://mlsanity.com/paper/1701.00495/) (2017)
- [Lip2AudSpec: Speech reconstruction from silent lip movements video](https://mlsanity.com/paper/1710.09798/) (2017)
- [Vocoder-Based Speech Synthesis from Silent Videos](https://mlsanity.com/paper/2004.02541/) (2020)
- [Lipreading with Long Short-Term Memory](https://mlsanity.com/paper/1601.08188/) (2016)
- [Silent Speech Interfaces for Speech Restoration: A Review](https://mlsanity.com/paper/2009.02110/) (2020)
- [Speech Synthesis using EEG](https://mlsanity.com/paper/2002.12756/) (2020)
- [Ultrasound-based Articulatory-to-Acoustic Mapping with WaveGlow Speech Synthesis](https://mlsanity.com/paper/2008.03152/) (2020)
- [Speaker dependent articulatory-to-acoustic mapping using real-time MRI of the vocal tract](https://mlsanity.com/paper/2008.00889/) (2020)

## Built on by (most cited first, 2 of 2)

- [Adaptation of Tacotron2-based Text-To-Speech for Articulatory-to-Acoustic Mapping using Ultrasound Tongue Imaging](https://mlsanity.com/paper/2107.12051/) (2021)
- [Towards Decoding Brain Activity During Passive Listening of Speech](https://mlsanity.com/paper/2402.16996/) (2024)

## Abstract (arXiv)

Articulatory-to-acoustic (forward) mapping is a technique to predict speech using various articulatory acquisition techniques as input (e.g. ultrasound tongue imaging, MRI, lip video). The advantage of lip video is that it is easily available and affordable: most modern smartphones have a front camera. There are already a few solutions for lip-to-speech synthesis, but they mostly concentrate on offline training and inference. In this paper, we propose a system built from a backend for deep neural network training and inference and a fronted as a form of a mobile application. Our initial evaluation shows that the scenario is feasible: a top-5 classification accuracy of 74% is combined with feedback from the mobile application user, making sure that the speaking impaired might be able to communicate with this solution.