# Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech

arXiv: https://arxiv.org/abs/2302.13700 · HTML: https://mlsanity.com/paper/2302.13700/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 27 Feb 2023.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 31)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [AdaSpeech: Adaptive Text to Speech for Custom Voice](https://mlsanity.com/paper/2103.00993/) (2021)
- [Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance](https://mlsanity.com/paper/2111.11755/) (2021)
- [NSML: Meet the MLaaS platform with a real-world case study](https://mlsanity.com/paper/1810.09957/) (2018)
- [Seeing Voices and Hearing Faces: Cross-modal biometric matching](https://mlsanity.com/paper/1804.00326/) (2018)
- [Learnable PINs: Cross-Modal Embeddings for Person Identity](https://mlsanity.com/paper/1805.00833/) (2018)
- [SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping](https://mlsanity.com/paper/2203.16749/) (2022)
- [Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data](https://mlsanity.com/paper/2205.15370/) (2022)
- [Disentangled Speech Embeddings using Cross-modal Self-supervision](https://mlsanity.com/paper/2002.08742/) (2020)
- [FaceFilter: Audio-visual speech separation using still images](https://mlsanity.com/paper/2005.07074/) (2020)
- [Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision](https://mlsanity.com/paper/2004.14326/) (2020)
- [Fitting New Speakers Based on a Short Untranscribed Sample](https://mlsanity.com/paper/1802.06984/) (2018)

## Built on by (most cited first, 27 of 27)

- [StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing](https://mlsanity.com/paper/2402.12636/) (2024)
- [Faces that Speak: Jointly Synthesising Talking Face and Speech from Text](https://mlsanity.com/paper/2405.10272/) (2024)
- [Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing](https://mlsanity.com/paper/2503.12042/) (2025)
- [MoEdit: On Learning Quantity Perception for Multi-object Image Editing](https://mlsanity.com/paper/2503.10112/) (2025)
- [Text-driven Talking Face Synthesis by Reprogramming Audio-driven Models](https://mlsanity.com/paper/2306.16003/) (2023)
- [FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing](https://mlsanity.com/paper/2505.01263/) (2025)
- [Hear Your Face: Face-based voice conversion with F0 estimation](https://mlsanity.com/paper/2408.09802/) (2024)
- [FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles](https://mlsanity.com/paper/2501.03181/) (2025)
- [Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion](https://mlsanity.com/paper/2409.00700/) (2024)
- [FunCineForge: A Unified Dataset Toolkit and Model for Zero-Shot Movie Dubbing in Diverse Cinematic Scenes](https://mlsanity.com/paper/2601.14777/) (2026)
- [Face-StyleSpeech: Enhancing Zero-shot Speech Synthesis from Face Images with Improved Face-to-Speech Mapping](https://mlsanity.com/paper/2311.05844/) (2023)
- [Facial Expression-Enhanced TTS: Combining Face Representation and Emotion Intensity for Adaptive Speech](https://mlsanity.com/paper/2409.16203/) (2024)
- [Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis](https://mlsanity.com/paper/2411.17690/) (2024)
- [A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning](https://mlsanity.com/paper/2412.00049/) (2024)
- [Unispeaker: A Unified Approach for Multimodality-driven Speaker Generation](https://mlsanity.com/paper/2501.06394/) (2025)
- [Realistic Speech-to-Face Generation with Speech-Conditioned Latent Diffusion Model with Face Prior](https://mlsanity.com/paper/2310.03363/) (2023)
- [Shushing! Let's Imagine an Authentic Speech from the Silent Video](https://mlsanity.com/paper/2503.14928/) (2025)
- [Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis](https://mlsanity.com/paper/2505.18972/) (2025)
- [TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation](https://mlsanity.com/paper/2512.20296/) (2025)
- [Towards Film-Making Production Dialogue, Narration, Monologue Adaptive Moving Dubbing Benchmarks](https://mlsanity.com/paper/2505.01450/) (2025)
- [MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbing](https://mlsanity.com/paper/2505.16279/) (2025)
- [Face2VoiceSync: Lightweight Face-Voice Consistency for Text-Driven Talking Face Generation](https://mlsanity.com/paper/2507.19225/) (2025)
- [Referee: Reference-aware Audiovisual Deepfake Detection](https://mlsanity.com/paper/2510.27475/) (2025)
- [Seeing What You Say: Expressive Image Generation from Speech](https://mlsanity.com/paper/2511.03423/) (2025)
- [Archon: A Unified Multimodal Model for Holistic Digital Human Generation](https://mlsanity.com/paper/2605.30311/) (2026)
- [Zero-Shot Face-to-Speech Synthesis via Latent Space Adaptation of a Style-Diffusion TTS Model](https://mlsanity.com/paper/2607.26742/) (2026)
- [Listen Before You Speak: Response Planning from Listener Facial Reactions for Conversational Speech Generation](https://mlsanity.com/paper/2609.21683/) (2026)

## Abstract (arXiv)

The goal of this work is zero-shot text-to-speech synthesis, with speaking styles and voices learnt from facial characteristics. Inspired by the natural fact that people can imagine the voice of someone when they look at his or her face, we introduce a face-styled diffusion text-to-speech (TTS) model within a unified framework learnt from visible attributes, called Face-TTS. This is the first time that face images are used as a condition to train a TTS model. We jointly train cross-model biometrics and TTS models to preserve speaker identity between face images and generated speech segments. We also propose a speaker feature binding loss to enforce the similarity of the generated and the ground truth speech segments in speaker embedding space. Since the biometric information is extracted directly from the face image, our method does not require extra fine-tuning steps to generate speech from unseen and unheard speakers. We train and evaluate the model on the LRS3 dataset, an in-the-wild audio-visual corpus containing background noise and diverse speaking styles. The project page is https://facetts.github.io.