Show Me Your Face, And I'll Tell You How You Speak
Not on the map. Why?
Abstract (arXiv)
When we speak, the prosody and content of the speech can be inferred from the movement of our lips. In this work, we explore the task of lip to speech synthesis, i.e., learning to generate speech given only the lip movements of a speaker where we focus on learning accurate lip to speech mappings for multiple speakers in unconstrained, large vocabulary settings. We capture the speaker's voice identity through their facial characteristics, i.e., age, gender, ethnicity and condition them along with the lip movements to generate speaker identity aware speech. To this end, we present a novel method "Lip2Speech", with key design choices to achieve accurate lip to speech synthesis in unconstrained scenarios. We also perform various experiments and extensive evaluation using quantitative, qualitative metrics and human evaluation.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 50 references that are in the map. First posted on arXiv 28 Jun 2022.
Builds on
35 of 50 references.
References · 35
- 2017Attention Is All You Needcited by 59,932
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
- 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
- 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
- 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
- 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
- 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
- 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
- 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
- 2014Learning Face Representation from Scratchcited by 451
23 more
- 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
- 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
- 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
- 2016Lip Reading Sentences in the Wildcited by 226
- 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
- 2018Neural Voice Cloning with a Few Samplescited by 100
- 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
- 2020Learning Individual Speaking Styles for Accurate Lip to Speech Synthesiscited by 74
- 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
- 2018Seeing Voices and Hearing Faces: Cross-modal biometric matchingcited by 60
- 2019Speech2Face: Learning the Face Behind a Voicecited by 55
- 2017Vid2speech: Speech Reconstruction from Silent Videocited by 53
- 2019Vision-Infused Deep Audio Inpaintingcited by 40
- 2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
- 2019Video-Driven Speech Reconstruction using Generative Adversarial Networkscited by 25
- 2018Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networkscited by 21
- 2019Lipper: Synthesizing Thy Speech using Multi-View Lipreadingcited by 19
- 2018Fitting New Speakers Based on a Short Untranscribed Samplecited by 18
- 2019Wav2Pix: Speech-conditioned Face Generation using Generative Adversarial Networkscited by 17
- 2017End-To-End Visual Speech Recognition With LSTMscited by 16
- 2016Lipreading with Long Short-Term Memorycited by 14
- 2018LCANet: End-to-End Lipreading with Cascaded Attention-CTCcited by 14
- 2021Facetron: A Multi-speaker Face-to-Speech Model based on Cross-modal Latent Representationscited by 4
Built on by
Not cited yet.