Controlled AutoEncoders to Generate Faces from Voices
Not on the map. Why?
Abstract (arXiv)
Multiple studies in the past have shown that there is a strong correlation between human vocal characteristics and facial features. However, existing approaches generate faces simply from voice, without exploring the set of features that contribute to these observed correlations. A computational methodology to explore this can be devised by rephrasing the question to: "how much would a target face have to change in order to be perceived as the originator of a source voice?" With this in perspective, we propose a framework to morph a target face in response to a given voice in a way that facial features are implicitly guided by learned voice-face correlation in this paper. Our framework includes a guided autoencoder that converts one face to another, controlled by a unique model-conditioning component called a gating controller which modifies the reconstructed face based on input voice recordings. We evaluate the framework on VoxCelab and VGGFace datasets through human subjects and face retrieval. Various experiments demonstrate the effectiveness of our proposed model.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 15 of its 26 references that are in the map. First posted on arXiv 16 Jul 2021.
Builds on
15 of 26 references.
References · 15
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
- 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
- 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
- 2018Pros and Cons of GAN Evaluation Measurescited by 224
- 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
- 2018Seeing Voices and Hearing Faces: Cross-modal biometric matchingcited by 60
- 2019HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Modelscited by 59
- 2019Speech2Face: Learning the Face Behind a Voicecited by 55
- 2017Adversarial Ranking for Language Generationcited by 50
- 2018On Learning Associations of Faces and Voicescited by 26
- 2018Disjoint Mapping Network for Cross-modal Matching of Voices and Facescited by 22
3 more
- 2018Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networkscited by 21
- 2018Identification of Deep Network Generated Images Using Disparities in Color Componentscited by 18
- 2019Wav2Pix: Speech-conditioned Face Generation using Generative Adversarial Networkscited by 17
Built on by
Citing papers per year: 2022: 1; 2023: 2.
3 papers.