Data to 7 Oct 2026

Vid2speech: Speech Reconstruction from Silent Video

arXiv 1701.00495 ↗First posted 2 Jan 2017cs.CV cs.SDcited by 53

Not on the map. Why?

Abstract (arXiv)

Speechreading is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible acoustic speech signal from silent video frames of a speaking person. The proposed CNN generates sound features for each frame based on its neighboring frames. Waveforms are then synthesized from the learned speech features to produce intelligible speech. We show that by leveraging the automatic feature learning capabilities of a CNN, we can obtain state-of-the-art word intelligibility on the GRID dataset, and show promising results for learning out-of-vocabulary (OOV) words.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Jan 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 20; 2023: 12; 2024: 5; 2025: 6; 2026: 1.

53 papers.

Sort:

By area:

2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 133
2022Visual Speech Recognition for Multiple Languages in the Wildcited by 94
2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
2022Lip to Speech Synthesis with Visual Context Attentional GANcited by 41
2021Neural Dubber: Dubbing for Videos According to Scriptscited by 39
2022SVTS: Scalable Video-to-Speech Synthesiscited by 35
2022Learning to Dub Movies via Hierarchical Prosody Modelscited by 28
2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27

38 more

2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24
2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
2023DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embeddingcited by 22
2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
2022Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Videocited by 18
2021Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generationcited by 14
2021More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speechcited by 12
2023Let There Be Sound: Reconstructing High Quality Speech from Silent Videoscited by 12
2025AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generationcited by 11
2021LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Readingcited by 10
2022FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesiscited by 8
2022Lip-to-Speech Synthesis for Arbitrary Speakers in the Wildcited by 8
2022ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancementcited by 8
2024Self-Supervised Audio-Visual Soundscape Stylizationcited by 8
2024Towards Accurate Lip-to-Speech Synthesis in-the-Wildcited by 7
2022VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversioncited by 6
2022VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selectioncited by 6
2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 6
2022Sound and Visual Representation Learning with Multiple Pretraining Taskscited by 5
2024V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flowcited by 5
20213D Convolutional Neural Networks for Ultrasound-Based Silent Speech Interfacescited by 4
2021Facetron: A Multi-speaker Face-to-Speech Model based on Cross-modal Latent Representationscited by 4
2025A Comprehensive Survey on Generative AI for Video-to-Music Generationcited by 4
2022Learning Speaker-specific Lip-to-Speech Generationcited by 3
2023Large-scale unsupervised audio pre-training for video-to-speech synthesiscited by 3
2023Speech Audio Synthesis from Tagged MRI and Non-Negative Matrix Factorization via Plastic Transformercited by 3
2025NaturalL2S: End-to-End High-quality Multispeaker Lip-to-Speech Synthesis with Differential Digital Signal Processingcited by 3
2025Sounding that Object: Interactive Object-Aware Image to Audio Generationcited by 3
2021Towards a practical lip-to-speech conversion system using deep neural networks and mobile application frontendcited by 2
2021Adaptation of Tacotron2-based Text-To-Speech for Articulatory-to-Acoustic Mapping using Ultrasound Tongue Imagingcited by 2
2022Is Lip Region-of-Interest Sufficient for Lipreading?cited by 1
2022Tagged-MRI Sequence to Audio Synthesis via Self Residual Attention Guided Heterogeneous Translatorcited by 1
2022Dual-Path Cross-Modal Attention for better Audio-Visual Speech Extractioncited by 1
2023Synthesizing audio from tongue motion during speech using tagged MRI via transformercited by 1
2023Adaptation of Tongue Ultrasound-Based Silent Speech Interfaces Using Spatial Transformer Networkscited by 1
2023RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representationscited by 1
2023Audio-visual video-to-speech synthesis with synthesized input audiocited by 1
2024Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRIcited by 1

50 most cited of 53 All 53 (JSON)