mlsanityData to 14 Oct 2026

SVTS: Scalable Video-to-Speech Synthesis

arXiv 2205.02058 ↗First posted 4 May 2022cs.SD cs.CV cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Video-to-speech synthesis (also known as lip-to-speech) refers to the translation of silent lip movements into the corresponding audio. This task has received an increasing amount of attention due to its self-supervised nature (i.e., can be trained without manual labelling) combined with the ever-growing collection of audio-visual data available online. Despite these strong motivations, contemporary video-to-speech works focus mainly on small- to medium-sized corpora with substantial constraints in both vocabulary and setting. In this work, we introduce a scalable video-to-speech framework consisting of two components: a video-to-spectrogram predictor and a pre-trained neural vocoder, which converts the mel-frequency spectrograms into waveform audio. We achieve state-of-the art results for GRID and considerably outperform previous approaches on LRW. More importantly, by focusing on spectrogram prediction using a simple feedforward model, we can efficiently and effectively scale our method to very large and unconstrained datasets: To the best of our knowledge, we are the first to show intelligible results on the challenging LRS3 dataset.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 4 May 2022.

Builds on

29 of 42 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  5. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  6. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  7. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  8. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  9. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 525
  10. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  11. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  12. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
17 more
  1. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  2. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  3. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  4. 2020Learning Individual Speaking Styles for Accurate Lip to Speech Synthesiscited by 74
  5. 2018End-to-end Audiovisual Speech Recognitioncited by 73
  6. 2020Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speechcited by 63
  7. 2017Vid2speech: Speech Reconstruction from Silent Videocited by 53
  8. 2022Lip to Speech Synthesis with Visual Context Attentional GANcited by 41
  9. 2017Lip2AudSpec: Speech reconstruction from silent lip movements videocited by 32
  10. 2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
  11. 2017Improved Speech Reconstruction from Silent Videocited by 26
  12. 2020StyleMelGAN: An Efficient High-Fidelity Adversarial Vocoder with Temporal Adaptive Normalizationcited by 26
  13. 2019Video-Driven Speech Reconstruction using Generative Adversarial Networkscited by 25
  14. 2020Vocoder-Based Speech Synthesis from Silent Videoscited by 20
  15. 2020Speech Prediction in Silent Videos using Variational Autoencoderscited by 18
  16. 2021Speaker disentanglement in video-to-speech conversioncited by 6
  17. 2021Facetron: A Multi-speaker Face-to-Speech Model based on Cross-modal Latent Representationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 9; 2024: 5; 2025: 13; 2026: 2.

35 papers.

Sort:

By area:

  1. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 47
  2. 2022Learning to Dub Movies via Hierarchical Prosody Modelscited by 29
  3. 2023Intelligible Lip-to-Speech Synthesis with Speech Unitscited by 28
  4. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  5. 2023DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embeddingcited by 22
  6. 2023Lip-to-Speech Synthesis in the Wild with Multi-task Learningcited by 21
  7. 2022Speaker-adaptive Lip Reading with User-dependent Paddingcited by 17
  8. 2023Let There Be Sound: Reconstructing High Quality Speech from Silent Videoscited by 12
  9. 2025AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generationcited by 12
  10. 2025VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Modelscited by 11
  11. 2025From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speechcited by 9
  12. 2022LA-VocE: Low-SNR Audio-visual Speech Enhancement using Neural Vocoderscited by 8
23 more
  1. 2022ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancementcited by 8
  2. 2024Towards Accurate Lip-to-Speech Synthesis in-the-Wildcited by 7
  3. 2024Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extractioncited by 7
  4. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 7
  5. 2024V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flowcited by 5
  6. 2025DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generationcited by 5
  7. 2025Supervising 3D Talking Head Avatars with Analysis-by-Audio-Synthesiscited by 5
  8. 2023Large-scale unsupervised audio pre-training for video-to-speech synthesiscited by 3
  9. 2025NaturalL2S: End-to-End High-quality Multispeaker Lip-to-Speech Synthesis with Differential Digital Signal Processingcited by 3
  10. 2023Zero-shot personalized lip-to-speech synthesis with face image based voice controlcited by 2
  11. 2024Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesiscited by 2
  12. 2023On the Audio-visual Synchronization for Lip-to-Speech Synthesiscited by 1
  13. 2023RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representationscited by 1
  14. 2023Audio-visual video-to-speech synthesis with synthesized input audiocited by 1
  15. 2025Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deploymentcited by 1
  16. 2025DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibilitycited by 1
  17. 2025Shushing! Let's Imagine an Authentic Speech from the Silent Videocited by 1
  18. 2025RESOUND: Speech Reconstruction from Silent Videos via Acoustic-Semantic Decomposed Modelingcited by 1
  19. 2025LTA-L2S: Lexical Tone-Aware Lip-to-Speech Synthesis for Mandarin with Cross-Lingual Transfer Learningcited by 1
  20. 2024Quantitative Analysis of Audio-Visual Tasks: An Information-Theoretic Perspectivecited by 0
  21. 2025MuteSwap: Visual-informed Silent Video Identity Conversioncited by 0
  22. 2026Cross-Modal Masking for Robust Silent Speech Synthesis Using sEMG and Lipreadingcited by 0
  23. 2026Watch Your Speech: Text-aware Video-to-Speech Synthesis with Textual Conditioningcited by 0