mlsanityData to 14 Oct 2026

Audio-visual video-to-speech synthesis with synthesized input audio

arXiv 2307.16584 ↗First posted 31 Jul 2023cs.SD cs.CV cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Video-to-speech synthesis involves reconstructing the speech signal of a speaker from a silent video. The implicit assumption of this task is that the sound signal is either missing or contains a high amount of noise/corruption such that it is not useful for processing. Previous works in the literature either use video inputs only or employ both video and audio inputs during training, and discard the input audio pathway during inference. In this work we investigate the effect of using video and audio inputs for video-to-speech synthesis during both training and inference. In particular, we use pre-trained video-to-speech models to synthesize the missing speech signals and then train an audio-visual-to-speech synthesis model, using both the silent video and the synthesized speech as inputs, to predict the final reconstructed speech. Our experiments demonstrate that this approach is successful with both raw waveforms and mel spectrograms as target outputs.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 58 of its 114 references that are in the map. First posted on arXiv 31 Jul 2023.

Builds on

58 of 114 references.

References · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2022Generative Adversarial Networkscited by 5,627
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  8. 2017Improved Training of Wasserstein GANscited by 2,235
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  12. 2016Least Squares Generative Adversarial Networkscited by 1,001
38 more
  1. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  2. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  3. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  4. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  5. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 525
  6. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  7. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  8. 2018Deep Audio-Visual Speech Recognitioncited by 322
  9. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  10. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
  11. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  12. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  13. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  14. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 216
  15. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  16. 2017S3FD: Single Shot Scale-invariant Face Detectorcited by 142
  17. 2017TasNet: time-domain audio separation network for real-time, single-channel speech separationcited by 115
  18. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  19. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 109
  20. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  21. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 81
  22. 2016LipNet: End-to-End Sentence-level Lipreadingcited by 80
  23. 2020Learning Individual Speaking Styles for Accurate Lip to Speech Synthesiscited by 74
  24. 2018End-to-end Audiovisual Speech Recognitioncited by 73
  25. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 68
  26. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 64
  27. 2018Large-Scale Visual Speech Recognitioncited by 57
  28. 2017Vid2speech: Speech Reconstruction from Silent Videocited by 53
  29. 2022Lip to Speech Synthesis with Visual Context Attentional GANcited by 41
  30. 2017Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networkscited by 36
  31. 2022SVTS: Scalable Video-to-Speech Synthesiscited by 35
  32. 2017Lip2AudSpec: Speech reconstruction from silent lip movements videocited by 32
  33. 2019Time Domain Audio Visual Speech Separationcited by 30
  34. 2019Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoderscited by 27
  35. 2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
  36. 2017Improved Speech Reconstruction from Silent Videocited by 26
  37. 2019Video-Driven Speech Reconstruction using Generative Adversarial Networkscited by 25
  38. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generationcited by 5