mlsanityData to 14 Oct 2026

Let There Be Sound: Reconstructing High Quality Speech from Silent Videos

arXiv 2308.15256 ↗First posted 29 Aug 2023eess.AS cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

The goal of this work is to reconstruct high quality speech from lip motions alone, a task also known as lip-to-speech. A key challenge of lip-to-speech systems is the one-to-many mapping caused by (1) the existence of homophenes and (2) multiple speech variations, resulting in a mispronounced and over-smoothed speech. In this paper, we propose a novel lip-to-speech system that significantly improves the generation quality by alleviating the one-to-many mapping problem from multiple perspectives. Specifically, we incorporate (1) self-supervised speech representations to disambiguate homophenes, and (2) acoustic variance information to model diverse speech styles. Additionally, to better solve the aforementioned problem, we employ a flow based post-net which captures and refines the details of the generated speech. We perform extensive experiments on two datasets, and demonstrate that our method achieves the generation quality close to that of real human utterance, outperforming existing methods in terms of speech naturalness and intelligibility by a large margin. Synthesised samples are available at our demo page: https://mm.kaist.ac.kr/projects/LTBS.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 45 of its 63 references that are in the map. First posted on arXiv 29 Aug 2023.

Builds on

45 of 63 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  9. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  10. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  12. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
33 more
  1. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 447
  2. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  3. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  4. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  5. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
  6. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  7. 2016Lip Reading Sentences in the Wildcited by 228
  8. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  9. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  10. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 151
  11. 2021Unsupervised Speech Recognitioncited by 148
  12. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  13. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 95
  14. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  15. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  16. 2020Learning Individual Speaking Styles for Accurate Lip to Speech Synthesiscited by 74
  17. 2018End-to-end Audiovisual Speech Recognitioncited by 73
  18. 2017Vid2speech: Speech Reconstruction from Silent Videocited by 53
  19. 2020Parallel Tacotron: Non-Autoregressive and Controllable TTScited by 49
  20. 2021PortaSpeech: Portable and High-Quality Generative Text-to-Speechcited by 47
  21. 2020Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesiscited by 41
  22. 2022Lip to Speech Synthesis with Visual Context Attentional GANcited by 41
  23. 2022Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Readingcited by 39
  24. 2022SVTS: Scalable Video-to-Speech Synthesiscited by 35
  25. 2022Revisiting Over-Smoothness in Text to Speechcited by 32
  26. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  27. 2020Multi-SpectroGAN: High-Diversity and High-Fidelity Spectrogram Generation with Adversarial Style Combination for Speech Synthesiscited by 28
  28. 2021Fre-GAN: Adversarial Frequency-consistent Audio Synthesiscited by 24
  29. 2023Lip-to-Speech Synthesis in the Wild with Multi-task Learningcited by 21
  30. 2019Lipper: Synthesizing Thy Speech using Multi-View Lipreadingcited by 19
  31. 2020Speech Prediction in Silent Videos using Variational Autoencoderscited by 18
  32. 2018Investigation of enhanced Tacotron text-to-speech synthesis systems with self-attention for pitch accent languagecited by 14
  33. 2022Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speechcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 4.

12 papers.

  1. 2025AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generationcited by 12
  2. 2025From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speechcited by 9
  3. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 7
  4. 2024V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flowcited by 5
  5. 2025Shushing! Let's Imagine an Authentic Speech from the Silent Videocited by 1
  6. 2025RESOUND: Speech Reconstruction from Silent Videos via Acoustic-Semantic Decomposed Modelingcited by 1
  7. 2025LTA-L2S: Lexical Tone-Aware Lip-to-Speech Synthesis for Mandarin with Cross-Lingual Transfer Learningcited by 1
  8. 2026CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbingcited by 1
  9. 2024CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generationcited by 0
  10. 2026SLD-L2S: Hierarchical Subspace Latent Diffusion for High-Fidelity Lip to Speech Synthesiscited by 0
  11. 2026Hierarchical Codec Diffusion for Video-to-Speech Generationcited by 0
  12. 2026Watch Your Speech: Text-aware Video-to-Speech Synthesis with Textual Conditioningcited by 0