# Polyglot: Multilingual Style Preserving Speech-Driven Facial Animation

arXiv: https://arxiv.org/abs/2604.16108 · HTML: https://mlsanity.com/paper/2604.16108/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 17 Apr 2026.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior](https://mlsanity.com/paper/2301.02379/) (2023)
- [GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents](https://mlsanity.com/paper/2303.14613/) (2023)
- [EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation](https://mlsanity.com/paper/2303.11089/) (2023)
- [Emotional Speech-Driven Animation with Content-Emotion Disentanglement](https://mlsanity.com/paper/2306.08990/) (2023)
- [Learning Formation of Physically-Based Face Attributes](https://mlsanity.com/paper/2004.03458/) (2020)
- [Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation](https://mlsanity.com/paper/1905.02876/) (2019)
- [Multiface: A Dataset for Neural Face Rendering](https://mlsanity.com/paper/2207.11243/) (2022)
- [DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models](https://mlsanity.com/paper/2310.00434/) (2023)
- [FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion](https://mlsanity.com/paper/2309.11306/) (2023)
- [Imitator: Personalized Speech-driven 3D Facial Animation](https://mlsanity.com/paper/2301.00023/) (2022)
- [SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Faces](https://mlsanity.com/paper/2306.10799/) (2023)
- [mHuBERT-147: A Compact Multilingual HuBERT Model](https://mlsanity.com/paper/2406.06371/) (2024)
- [TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudio](https://mlsanity.com/paper/2304.01448/) (2023)
- [UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Model](https://mlsanity.com/paper/2408.00762/) (2024)
- [FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2303.05416/) (2023)
- [ScanTalk: 3D Talking Heads from Unregistered Scans](https://mlsanity.com/paper/2403.10942/) (2024)
- [ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performer](https://mlsanity.com/paper/2408.03284/) (2024)
- [3DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editing](https://mlsanity.com/paper/2312.00870/) (2023)
- [PersonaTalk: Bring Attention to Your Persona in Visual Dubbing](https://mlsanity.com/paper/2409.05379/) (2024)
- [EmoVOCA: Speech-Driven Emotional 3D Talking Heads](https://mlsanity.com/paper/2403.12886/) (2024)
- [ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAE](https://mlsanity.com/paper/2409.07966/) (2024)
- [A Sparse and Locally Coherent Morphable Face Model for Dense Semantic Correspondence Across Heterogeneous 3D Faces](https://mlsanity.com/paper/2006.03840/) (2020)
- [Learning Landmarks Motion from Speech for Speaker-Agnostic 3D Talking Heads Generation](https://mlsanity.com/paper/2306.01415/) (2023)
- [Beyond Fixed Topologies: Unregistered Training and Comprehensive Evaluation Metrics for 3D Talking Heads](https://mlsanity.com/paper/2410.11041/) (2024)

## Abstract (arXiv)

Speech-Driven Facial Animation (SDFA) has gained significant attention due to its applications in movies, video games, and virtual reality. However, most existing models are trained on single-language data, limiting their effectiveness in real-world multilingual scenarios. In this work, we address multilingual SDFA, which is essential for realistic generation since language influences phonetics, rhythm, intonation, and facial expressions. Speaking style is also shaped by individual differences, not only by language. Existing methods typically rely on either language-specific or speaker-specific conditioning, but not both, limiting their ability to model their interaction. We introduce Polyglot, a unified diffusion-based architecture for personalized multilingual SDFA. Our method uses transcript embeddings to encode language information and style embeddings extracted from reference facial sequences to capture individual speaking characteristics. Polyglot does not require predefined language or speaker labels, enabling generalization across languages and speakers through self-supervised learning. By jointly conditioning on language and style, it captures expressive traits such as rhythm, articulation, and habitual facial movements, producing temporally coherent and realistic animations. Experiments show improved performance in both monolingual and multilingual settings, providing a unified framework for modeling language and personal style in SDFA.