# Xemo-Talker: Unlock Emotions Explicitly for Audio-Driven Talking Portrait Synthesis

arXiv: https://arxiv.org/abs/2608.14700 · HTML: https://mlsanity.com/paper/2608.14700/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://mlsanity.com/paper/2407.03168/) (2024)
- [EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model](https://mlsanity.com/paper/2205.15278/) (2022)
- [Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer](https://mlsanity.com/paper/2412.00733/) (2024)
- [Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation](https://mlsanity.com/paper/2410.07718/) (2024)
- [Efficient Emotional Adaptation for Audio-Driven Talking-Head Generation](https://mlsanity.com/paper/2309.04946/) (2023)
- [EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis](https://mlsanity.com/paper/2404.01647/) (2024)
- [FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis](https://mlsanity.com/paper/2504.04842/) (2025)
- [SPACE: Speech-driven Portrait Animation with Controllable Expression](https://mlsanity.com/paper/2211.09809/) (2022)
- [MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation](https://mlsanity.com/paper/2412.04448/) (2024)
- [EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion](https://mlsanity.com/paper/2411.16726/) (2024)
- [SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers](https://mlsanity.com/paper/2506.00830/) (2025)
- [Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generation](https://mlsanity.com/paper/2504.18087/) (2025)
- [PC-Talk: Precise Facial Animation Control for Audio-Driven Talking Face Generation](https://mlsanity.com/paper/2503.14295/) (2025)
- [Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based Spatiotemporal Diffusion for Audio-driven Talking Portrait](https://mlsanity.com/paper/2503.12963/) (2025)
- [EmoHead: Emotional Talking Head via Manipulating Semantic Expression Parameters](https://mlsanity.com/paper/2503.19416/) (2025)
- [GMTalker: Gaussian Mixture-based Audio-Driven Emotional Talking Video Portraits](https://mlsanity.com/paper/2312.07669/) (2023)

## Abstract (arXiv)

Precise emotion control in audio-driven talking heads remains a challenge due to the reliance on implicit emotion regulation in existing systems, which often leads to indirect and insufficient control. Additionally, training with explicit emotion-related losses across the entire motion space poses significant difficulties due to the inherent trade-off between accurate lip synchronization and fine-grained emotion control. In this paper, we reveal a key finding: although emotional cues are distributed throughout the motion space, concentrating discriminative supervision on less-principal components achieves a better emotion-lip synchronization balance, as principal components mainly encode high-energy articulation and pose variations. Building on this insight, we propose Xemo-Talker, which first learns a neutral speech-to-motion mapping for stable articulation and lip synchronization, and then introduces a lightweight emotion branch guided by less-principal subspace supervision. To enhance emotion control, we design a Tri-Loss consisting of inter-class separation, intra-class compactness, and less-principal contrastive learning. Given an audio input, a reference image, and an emotion label, Xemo-Talker achieves state-of-the-art emotion classification accuracy while maintaining competitive lip synchronization and high inference efficiency, with performance approaching that measured on real videos.The source code is publicly available at https://github.com/chaolongy/Xemo-Talker.