mlsanityData to 7 Oct 2026

Talking Face Generation with Multilingual TTS

arXiv 2205.06421 ↗First posted 13 May 2022cs.CV cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

In this work, we propose a joint system combining a talking face generation system with a text-to-speech system that can generate multilingual talking face videos from only the text input. Our system can synthesize natural multilingual speeches while maintaining the vocal identity of the speaker, as well as lip movements synchronized to the synthesized speech. We demonstrate the generalization capabilities of our system by selecting four languages (Korean, English, Japanese, and Chinese) each from a different language family. We also compare the outputs of our talking face generation model to outputs of a prior work that claims multilingual support. For our demo, we add a translation API to the preprocessing stage and present it in the form of a neural dubber so that users can utilize the multilingual property of our system more easily.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 16 of its 20 references that are in the map. First posted on arXiv 13 May 2022.

Builds on

16 of 20 references.

References · 16

  1. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  2. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  4. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  5. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  6. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  7. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  8. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  9. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  10. 2020Towards Fast, Accurate and Stable 3D Dense Face Alignmentcited by 219
  11. 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
  12. 2019Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloningcited by 66
4 more
  1. 2021WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesiscited by 65
  2. 2017JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesiscited by 57
  3. 2019JVS corpus: free Japanese multi-speaker voice corpuscited by 20
  4. 2020Generating Multilingual Voices Using Speaker Space Translation Based on Bilingual Speaker Datacited by 7

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 7; 2024: 4; 2025: 5.

18 papers.

Sort:

By area:

  1. 2023FT2TF: First-Person Statement Text-To-Talking Face Generationcited by 26
  2. 2023MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognitioncited by 14
  3. 2024JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generationcited by 9
  4. 2025AV-Flow: Transforming Text to Audio-Visual Human-like Interactionscited by 9
  5. 2025OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimickingcited by 9
  6. 2024PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generationcited by 5
  7. 2022Synthesizing Photorealistic Virtual Humans Through Cross-modal Disentanglementcited by 4
  8. 2023TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translationcited by 4
  9. 2024Stereo-Talker: Audio-driven 3D Human Synthesis with Prior-Guided Mixture-of-Expertscited by 4
  10. 2023Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realismcited by 3
  11. 2024JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Modelcited by 3
  12. 2025Unmasking Puppeteers: Leveraging Biometric Leakage to Expose Impersonation in AI-Based Videoconferencingcited by 2
6 more
  1. 2025MAGIC-Talk: Motion-aware Audio-Driven Talking Face Generation with Customizable Identity Controlcited by 2
  2. 2022IMaSC -- ICFOSS Malayalam Speech Corpuscited by 1
  3. 2023A Unified Compression Framework for Efficient Speech-Driven Talking-Face Generationcited by 0
  4. 2023CPNet: Exploiting CLIP-based Attention Condenser and Probability Map Guidance for High-fidelity Talking Face Generationcited by 0
  5. 2023FACTS: Facial Animation Creation using the Transfer of Stylescited by 0
  6. 2025Generative AI for Video Translation: A Scalable Architecture for Multilingual Video Conferencingcited by 0