# Vid2speech: Speech Reconstruction from Silent Video

arXiv: https://arxiv.org/abs/1701.00495 · HTML: https://mlsanity.com/paper/1701.00495/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2017.

## Built on by (most cited first, 30 of 53)

- [Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2306.17203/) (2023)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation](https://mlsanity.com/paper/2401.04747/) (2024)
- [Conditional Generation of Audio from Video via Foley Analogies](https://mlsanity.com/paper/2304.08490/) (2023)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Lip to Speech Synthesis with Visual Context Attentional GAN](https://mlsanity.com/paper/2204.01726/) (2022)
- [Neural Dubber: Dubbing for Videos According to Scripts](https://mlsanity.com/paper/2110.08243/) (2021)
- [SVTS: Scalable Video-to-Speech Synthesis](https://mlsanity.com/paper/2205.02058/) (2022)
- [Learning to Dub Movies via Hierarchical Prosody Models](https://mlsanity.com/paper/2212.04054/) (2022)
- [End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks](https://mlsanity.com/paper/2104.13332/) (2021)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis](https://mlsanity.com/paper/2203.17263/) (2022)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embedding](https://mlsanity.com/paper/2308.07787/) (2023)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video](https://mlsanity.com/paper/2204.01265/) (2022)
- [Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generation](https://mlsanity.com/paper/2105.00708/) (2021)
- [More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speech](https://mlsanity.com/paper/2111.10139/) (2021)
- [Let There Be Sound: Reconstructing High Quality Speech from Silent Videos](https://mlsanity.com/paper/2308.15256/) (2023)
- [AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generation](https://mlsanity.com/paper/2508.00733/) (2025)
- [LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading](https://mlsanity.com/paper/2112.04748/) (2021)
- [FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis](https://mlsanity.com/paper/2207.03800/) (2022)
- [Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild](https://mlsanity.com/paper/2209.00642/) (2022)
- [ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement](https://mlsanity.com/paper/2212.11377/) (2022)
- [Self-Supervised Audio-Visual Soundscape Stylization](https://mlsanity.com/paper/2409.14340/) (2024)
- [Towards Accurate Lip-to-Speech Synthesis in-the-Wild](https://mlsanity.com/paper/2403.01087/) (2024)
- [AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation](https://mlsanity.com/paper/2504.20629/) (2025)
- [VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion](https://mlsanity.com/paper/2202.09081/) (2022)
- [VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selection](https://mlsanity.com/paper/2206.07458/) (2022)

## Abstract (arXiv)

Speechreading is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible acoustic speech signal from silent video frames of a speaking person. The proposed CNN generates sound features for each frame based on its neighboring frames. Waveforms are then synthesized from the learned speech features to produce intelligible speech. We show that by leveraging the automatic feature learning capabilities of a CNN, we can obtain state-of-the-art word intelligibility on the GRID dataset, and show promising results for learning out-of-vocabulary (OOV) words.