mlsanityData to 7 Oct 2026

Improved Speech Reconstruction from Silent Video

arXiv 1708.01204 ↗First posted 1 Aug 2017cs.CV cs.SDcited by 26

Not on the map. Why?

Abstract (arXiv)

Speechreading is the task of inferring phonetic information from visually observed articulatory facial movements, and is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible and natural-sounding acoustic speech signal from silent video frames of a speaking person. We train our model on speakers from the GRID and TCD-TIMIT datasets, and evaluate the quality and intelligibility of reconstructed speech using common objective measurements. We show that speech predictions from the proposed model attain scores which indicate significantly improved quality over existing models. In addition, we show promising results towards reconstructing speech from an unconstrained dictionary.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Aug 2017.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 13; 2023: 6; 2024: 1.

26 papers.

Sort:

  1. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  2. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  3. 2022Lip to Speech Synthesis with Visual Context Attentional GANcited by 41
  4. 2022SVTS: Scalable Video-to-Speech Synthesiscited by 35
  5. 2022Learning to Dub Movies via Hierarchical Prosody Modelscited by 28
  6. 2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
  7. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24
  8. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  9. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  10. 2022Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Videocited by 18
  11. 2021Decoding EEG Brain Activity for Multi-Modal Natural Language Processingcited by 13
  12. 2021LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Readingcited by 10
14 more
  1. 2022FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesiscited by 8
  2. 2022Lip-to-Speech Synthesis for Arbitrary Speakers in the Wildcited by 8
  3. 2024Towards Accurate Lip-to-Speech Synthesis in-the-Wildcited by 7
  4. 2022VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversioncited by 6
  5. 2022VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selectioncited by 6
  6. 2023SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networkscited by 5
  7. 2022Learning Speaker-specific Lip-to-Speech Generationcited by 3
  8. 2023Large-scale unsupervised audio pre-training for video-to-speech synthesiscited by 3
  9. 2021Audio-Visual Speech Separation Using Cross-Modal Correspondence Losscited by 2
  10. 2021Improving Ultrasound Tongue Image Reconstruction from Lip Images Using Self-supervised Learning and Attention Mechanismcited by 2
  11. 2023On the Audio-visual Synchronization for Lip-to-Speech Synthesiscited by 1
  12. 2023RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representationscited by 1
  13. 2023Audio-visual video-to-speech synthesis with synthesized input audiocited by 1
  14. 2021Multi-Modulation Network for Audio-Visual Event Localizationcited by 0