# Wav2Pix: Speech-conditioned Face Generation using Generative Adversarial Networks

arXiv: https://arxiv.org/abs/1903.10195 · HTML: https://mlsanity.com/paper/1903.10195/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2019.

## Built on by (most cited first, 17 of 17)

- [SPACE: Speech-driven Portrait Animation with Controllable Expression](https://mlsanity.com/paper/2211.09809/) (2022)
- [Conditional Vector Graphics Generation for Music Cover Images](https://mlsanity.com/paper/2205.07301/) (2022)
- [Gaussian Harmony: Attaining Fairness in Diffusion-based Face Generation Models](https://mlsanity.com/paper/2312.14976/) (2023)
- [Controlled AutoEncoders to Generate Faces from Voices](https://mlsanity.com/paper/2107.07988/) (2021)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion](https://mlsanity.com/paper/2409.00700/) (2024)
- [Fighting Malicious Media Data: A Survey on Tampering Detection and Deepfake Detection](https://mlsanity.com/paper/2212.05667/) (2022)
- [Fusion-S2iGan: An Efficient and Effective Single-Stage Framework for Speech-to-Image Generation](https://mlsanity.com/paper/2305.10126/) (2023)
- [MACS: Multi-source Audio-to-image Generation with Contextual Significance and Semantic Alignment](https://mlsanity.com/paper/2503.10287/) (2025)
- [Cross-Modal Virtual Sensing for Combustion Instability Monitoring](https://mlsanity.com/paper/2110.01659/) (2021)
- [The applicability of transperceptual and deep learning approaches to the study and mimicry of complex cartilaginous tissues](https://mlsanity.com/paper/2211.14314/) (2022)
- [FTFDNet: Learning to Detect Talking Face Video Manipulation with Tri-Modality Interaction](https://mlsanity.com/paper/2307.03990/) (2023)
- [Realistic Speech-to-Face Generation with Speech-Conditioned Latent Diffusion Model with Face Prior](https://mlsanity.com/paper/2310.03363/) (2023)
- [Identity-guided Face Generation with Multi-modal Contour Conditions](https://mlsanity.com/paper/2110.04854/) (2021)
- [Show Me Your Face, And I'll Tell You How You Speak](https://mlsanity.com/paper/2206.14009/) (2022)
- [Audio Input Generates Continuous Frames to Synthesize Facial Video Using Generative Adiversarial Networks](https://mlsanity.com/paper/2207.08813/) (2022)
- [See the Speaker: Crafting High-Resolution Talking Faces from Speech with Prior Guidance and Region Refinement](https://mlsanity.com/paper/2510.26819/) (2025)

## Abstract (arXiv)

Speech is a rich biometric signal that contains information about the identity, gender and emotional state of the speaker. In this work, we explore its potential to generate face images of a speaker by conditioning a Generative Adversarial Network (GAN) with raw speech input. We propose a deep neural network that is trained from scratch in an end-to-end fashion, generating a face directly from the raw speech waveform without any additional identity information (e.g reference image or one-hot encoding). Our model is trained in a self-supervised approach by exploiting the audio and visual signals naturally aligned in videos. With the purpose of training from video data, we present a novel dataset collected for this work, with high-quality videos of youtubers with notable expressiveness in both the speech and visual signals.