# Video-Driven Speech Reconstruction using Generative Adversarial Networks

arXiv: https://arxiv.org/abs/1906.06301 · HTML: https://mlsanity.com/paper/1906.06301/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2019.

## Built on by (most cited first, 25 of 25)

- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Lip to Speech Synthesis with Visual Context Attentional GAN](https://mlsanity.com/paper/2204.01726/) (2022)
- [SVTS: Scalable Video-to-Speech Synthesis](https://mlsanity.com/paper/2205.02058/) (2022)
- [End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks](https://mlsanity.com/paper/2104.13332/) (2021)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embedding](https://mlsanity.com/paper/2308.07787/) (2023)
- [Lip-to-Speech Synthesis in the Wild with Multi-task Learning](https://mlsanity.com/paper/2302.08841/) (2023)
- [Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video](https://mlsanity.com/paper/2204.01265/) (2022)
- [LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading](https://mlsanity.com/paper/2112.04748/) (2021)
- [FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis](https://mlsanity.com/paper/2207.03800/) (2022)
- [Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild](https://mlsanity.com/paper/2209.00642/) (2022)
- [Towards Accurate Lip-to-Speech Synthesis in-the-Wild](https://mlsanity.com/paper/2403.01087/) (2024)
- [VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion](https://mlsanity.com/paper/2202.09081/) (2022)
- [VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selection](https://mlsanity.com/paper/2206.07458/) (2022)
- [Supervising 3D Talking Head Avatars with Analysis-by-Audio-Synthesis](https://mlsanity.com/paper/2504.13386/) (2025)
- [Facetron: A Multi-speaker Face-to-Speech Model based on Cross-modal Latent Representations](https://mlsanity.com/paper/2107.12003/) (2021)
- [Learning Speaker-specific Lip-to-Speech Generation](https://mlsanity.com/paper/2206.02050/) (2022)
- [Large-scale unsupervised audio pre-training for video-to-speech synthesis](https://mlsanity.com/paper/2306.15464/) (2023)
- [DINO: A Conditional Energy-Based GAN for Domain Translation](https://mlsanity.com/paper/2102.09281/) (2021)
- [Zero-shot personalized lip-to-speech synthesis with face image based voice control](https://mlsanity.com/paper/2305.14359/) (2023)
- [On the Audio-visual Synchronization for Lip-to-Speech Synthesis](https://mlsanity.com/paper/2303.00502/) (2023)
- [RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representations](https://mlsanity.com/paper/2307.01233/) (2023)
- [Audio-visual video-to-speech synthesis with synthesized input audio](https://mlsanity.com/paper/2307.16584/) (2023)
- [Show Me Your Face, And I'll Tell You How You Speak](https://mlsanity.com/paper/2206.14009/) (2022)

## Abstract (arXiv)

Speech is a means of communication which relies on both audio and visual information. The absence of one modality can often lead to confusion or misinterpretation of information. In this paper we present an end-to-end temporal model capable of directly synthesising audio from silent video, without needing to transform to-and-from intermediate features. Our proposed approach, based on GANs is capable of producing natural sounding, intelligible speech which is synchronised with the video. The performance of our model is evaluated on the GRID dataset for both speaker dependent and speaker independent scenarios. To the best of our knowledge this is the first method that maps video directly to raw audio and the first to produce intelligible speech when tested on previously unseen speakers. We evaluate the synthesised audio not only based on the sound quality but also on the accuracy of the spoken words.