# Animating Face using Disentangled Audio Representations

arXiv: https://arxiv.org/abs/1910.00726 · HTML: https://mlsanity.com/paper/1910.00726/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2019.

## Built on by (most cited first, 24 of 24)

- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [Audio-Driven Emotional Video Portraits](https://mlsanity.com/paper/2104.07452/) (2021)
- [EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model](https://mlsanity.com/paper/2205.15278/) (2022)
- [EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation](https://mlsanity.com/paper/2303.11089/) (2023)
- [Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert](https://mlsanity.com/paper/2303.17480/) (2023)
- [LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalization](https://mlsanity.com/paper/2106.04185/) (2021)
- [Synthetic Data -- what, why and how?](https://mlsanity.com/paper/2205.03257/) (2022)
- [SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory](https://mlsanity.com/paper/2211.00924/) (2022)
- [Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors](https://mlsanity.com/paper/2212.04248/) (2022)
- [Speech Driven Video Editing via an Audio-Conditioned Diffusion Model](https://mlsanity.com/paper/2301.04474/) (2023)
- [Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animation](https://mlsanity.com/paper/2112.02214/) (2021)
- [Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications](https://mlsanity.com/paper/2311.18168/) (2023)
- [Speech2Video: Cross-Modal Distillation for Speech to Video Generation](https://mlsanity.com/paper/2107.04806/) (2021)
- [T3M: Text Guided 3D Human Motion Synthesis from Speech](https://mlsanity.com/paper/2408.12885/) (2024)
- [A Comprehensive Taxonomy and Analysis of Talking Head Synthesis: Techniques for Portrait Generation, Driving Mechanisms, and Editing](https://mlsanity.com/paper/2406.10553/) (2024)
- [Style Transfer for 2D Talking Head Animation](https://mlsanity.com/paper/2303.09799/) (2023)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models](https://mlsanity.com/paper/2602.07106/) (2026)
- [Personalized Speech-driven Expressive 3D Facial Animation Synthesis with Style Control](https://mlsanity.com/paper/2310.17011/) (2023)
- [Disentanglement Beyond Static vs. Dynamic: A Benchmark and Evaluation Framework for Multi-Factor Sequential Representations](https://mlsanity.com/paper/2510.17313/) (2025)
- [LARNet: Latent Action Representation for Human Action Synthesis](https://mlsanity.com/paper/2110.10899/) (2021)
- [Facial Landmark Predictions with Applications to Metaverse](https://mlsanity.com/paper/2209.14698/) (2022)
- [Towards Unified Co-Speech Gesture Generation via Hierarchical Implicit Periodicity Learning](https://mlsanity.com/paper/2512.13131/) (2025)
- [CIViC: Evidence Gates for Audio-Conditioned Visual Prediction on Proxy Tasks](https://mlsanity.com/paper/2610.15327/) (2026)

## Abstract (arXiv)

All previous methods for audio-driven talking head generation assume the input audio to be clean with a neutral tone. As we show empirically, one can easily break these systems by simply adding certain background noise to the utterance or changing its emotional tone (to such as sad). To make talking head generation robust to such variations, we propose an explicit audio representation learning framework that disentangles audio sequences into various factors such as phonetic content, emotional tone, background noise and others. We conduct experiments to validate that conditioned on disentangled content representation, the generated mouth movement by our model is significantly more accurate than previous approaches (without disentangled learning) in the presence of noise and emotional variations. We further demonstrate that our framework is compatible with current state-of-the-art approaches by replacing their original audio learning component with ours. To our best knowledge, this is the first work which improves the performance of talking head generation from disentangled audio representation perspective, which is important for many real-world applications.