mlsanityData to 7 Oct 2026

Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech

arXiv 2302.13700 ↗First posted 27 Feb 2023cs.LG cs.CV cs.SDcited by 27

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

The goal of this work is zero-shot text-to-speech synthesis, with speaking styles and voices learnt from facial characteristics. Inspired by the natural fact that people can imagine the voice of someone when they look at his or her face, we introduce a face-styled diffusion text-to-speech (TTS) model within a unified framework learnt from visible attributes, called Face-TTS. This is the first time that face images are used as a condition to train a TTS model. We jointly train cross-model biometrics and TTS models to preserve speaker identity between face images and generated speech segments. We also propose a speaker feature binding loss to enforce the similarity of the generated and the ground truth speech segments in speaker embedding space. Since the biometric information is extracted directly from the face image, our method does not require extra fine-tuning steps to generate speech from unseen and unheard speakers. We train and evaluate the model on the LRS3 dataset, an in-the-wild audio-visual corpus containing background noise and diverse speaking styles. The project page is https://facetts.github.io.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 31 of its 37 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

31 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data30 May
  2. 2022SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping31 Mar
  3. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance23 Nov
  4. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  5. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 26

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  9. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  10. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  11. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  12. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
14 more
  1. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  2. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  3. 2018Deep Audio-Visual Speech Recognitioncited by 318
  4. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  5. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  6. 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
  7. 2018NSML: Meet the MLaaS platform with a real-world case studycited by 70
  8. 2018Seeing Voices and Hearing Faces: Cross-modal biometric matchingcited by 60
  9. 2018Learnable PINs: Cross-Modal Embeddings for Person Identitycited by 60
  10. 2020Disentangled Speech Embeddings using Cross-modal Self-supervisioncited by 30
  11. 2020FaceFilter: Audio-visual speech separation using still imagescited by 25
  12. 2020Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervisioncited by 19
  13. 2018Fitting New Speakers Based on a Short Untranscribed Samplecited by 18
  14. 2022Residual-guided Personalized Speech Synthesis based on Face Imagecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 13; 2026: 4.

27 papers.

Sort:

By area:

  1. 2024Faces that Speak: Jointly Synthesising Talking Face and Speech from Textcited by 21
  2. 2024StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbingcited by 20
  3. 2025Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbingcited by 9
  4. 2025MoEdit: On Learning Quantity Perception for Multi-object Image Editingcited by 7
  5. 2023Text-driven Talking Face Synthesis by Reprogramming Audio-driven Modelscited by 6
  6. 2025FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancingcited by 6
  7. 2024Hear Your Face: Face-based voice conversion with F0 estimationcited by 4
  8. 2025FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Stylescited by 4
  9. 2024Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversioncited by 3
  10. 2026FunCineForge: A Unified Dataset Toolkit and Model for Zero-Shot Movie Dubbing in Diverse Cinematic Scenescited by 3
  11. 2023Face-StyleSpeech: Enhancing Zero-shot Speech Synthesis from Face Images with Improved Face-to-Speech Mappingcited by 2
  12. 2024Facial Expression-Enhanced TTS: Combining Face Representation and Emotion Intensity for Adaptive Speechcited by 2
15 more
  1. 2024Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesiscited by 2
  2. 2024A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learningcited by 2
  3. 2025Unispeaker: A Unified Approach for Multimodality-driven Speaker Generationcited by 2
  4. 2023Realistic Speech-to-Face Generation with Speech-Conditioned Latent Diffusion Model with Face Priorcited by 1
  5. 2025Shushing! Let's Imagine an Authentic Speech from the Silent Videocited by 1
  6. 2025Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesiscited by 1
  7. 2025TAVID: Text-Driven Audio-Visual Interactive Dialogue Generationcited by 1
  8. 2025Towards Film-Making Production Dialogue, Narration, Monologue Adaptive Moving Dubbing Benchmarkscited by 0
  9. 2025MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbingcited by 0
  10. 2025Face2VoiceSync: Lightweight Face-Voice Consistency for Text-Driven Talking Face Generationcited by 0
  11. 2025Referee: Reference-aware Audiovisual Deepfake Detectioncited by 0
  12. 2025Seeing What You Say: Expressive Image Generation from Speechcited by 0
  13. 2026Archon: A Unified Multimodal Model for Holistic Digital Human Generationcited by 0
  14. 2026Zero-Shot Face-to-Speech Synthesis via Latent Space Adaptation of a Style-Diffusion TTS Modelcited by 0
  15. 2026Listen Before You Speak: Response Planning from Listener Facial Reactions for Conversational Speech Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusion
  2. 2025From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech
  3. 2026VoxMorph: Scalable Zero-shot Voice Identity Morphing via Disentangled Embeddings
  4. 2025Emotional Face-to-Speech
  5. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias
  6. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  7. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
  8. 2025Text-Driven Voice Conversion via Latent State-Space Modeling ↗