mlsanityData to 14 Oct 2026

Shared Latent Representation for Joint Text-to-Audio-Visual Synthesis

arXiv 2511.05432 ↗First posted 7 Nov 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose a text-to-talking-face synthesis framework leveraging latent speech representations from HierSpeech++. A Text-to-Vec module generates Wav2Vec2 embeddings from text, which jointly condition speech and face generation. To handle distribution shifts between clean and TTS-predicted features, we adopt a two-stage training: pretraining on Wav2Vec2 embeddings and finetuning on TTS outputs. This enables tight audio-visual alignment, preserves speaker identity, and produces natural, expressive speech and synchronized facial motion without ground-truth audio at inference. Experiments show that conditioning on TTS-predicted latent features outperforms cascaded pipelines, improving both lip-sync and visual realism.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 7 Nov 2025.

Builds on

22 of 27 references.

References · 22

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  4. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  5. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  6. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  7. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  8. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  9. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 98
  10. 2023Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronizationcited by 49
  11. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  12. 2021Text2Video: Text-driven Talking-head Video Synthesis with Personalized Phoneme-Pose Dictionarycited by 27
10 more
  1. 2023FT2TF: First-Person Statement Text-To-Talking Face Generationcited by 26
  2. 2024Faces that Speak: Jointly Synthesising Talking Face and Speech from Textcited by 21
  3. 2022Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of Videoscited by 18
  4. 2024Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluationcited by 17
  5. 2023Audio-driven Talking Face Generation with Stabilized Synchronization Losscited by 13
  6. 2025OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimickingcited by 9
  7. 2023UniFLG: Unified Facial Landmark Generator from Text or Speechcited by 8
  8. 2023Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesiscited by 8
  9. 2023Text-to-Video: a Two-stage Framework for Zero-shot Identity-agnostic Talking-head Generationcited by 8
  10. 2023Text-driven Talking Face Synthesis by Reprogramming Audio-driven Modelscited by 6

Built on by

Not cited yet.