# EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars

arXiv: https://arxiv.org/abs/2404.19110 · HTML: https://mlsanity.com/paper/2404.19110/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 30 of 32)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [I M Avatar: Implicit Morphable Head Avatars from Videos](https://mlsanity.com/paper/2112.07471/) (2021)
- [Neural Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2112.01554/) (2021)
- [StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN](https://mlsanity.com/paper/2203.04036/) (2022)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [MegaPortraits: One-shot Megapixel Neural Head Avatars](https://mlsanity.com/paper/2207.07621/) (2022)
- [Realistic One-shot Mesh-based Head Avatars](https://mlsanity.com/paper/2206.08343/) (2022)
- [Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatars](https://mlsanity.com/paper/2008.10174/) (2020)
- [Neural Head Reenactment with Latent Pose Descriptors](https://mlsanity.com/paper/2004.12000/) (2020)
- [AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxels](https://mlsanity.com/paper/2211.13206/) (2022)
- [MetaPortrait: Identity-Preserving Talking Head Generation with Fast Personalized Adaptation](https://mlsanity.com/paper/2212.08062/) (2022)
- [NOFA: NeRF-based One-shot Facial Avatar Reconstruction](https://mlsanity.com/paper/2307.03441/) (2023)
- [SPACE: Speech-driven Portrait Animation with Controllable Expression](https://mlsanity.com/paper/2211.09809/) (2022)
- [HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces](https://mlsanity.com/paper/2307.10797/) (2023)
- [Understanding Collapse in Non-Contrastive Siamese Representation Learning](https://mlsanity.com/paper/2209.15007/) (2022)
- [Generalizable One-shot Neural Head Avatar](https://mlsanity.com/paper/2306.08768/) (2023)

## Built on by (most cited first, 30 of 66)

- [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://mlsanity.com/paper/2407.03168/) (2024)
- [Wan-Animate: Unified Character Animation and Replacement with Holistic Replication](https://mlsanity.com/paper/2509.14055/) (2025)
- [Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis](https://mlsanity.com/paper/2411.19509/) (2024)
- [FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait](https://mlsanity.com/paper/2412.01064/) (2024)
- [DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidance](https://mlsanity.com/paper/2504.01724/) (2025)
- [X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention](https://mlsanity.com/paper/2507.23143/) (2025)
- [Human Motion Video Generation: A Survey](https://mlsanity.com/paper/2509.03883/) (2025)
- [Pippo: High-Resolution Multi-View Humans from a Single Image](https://mlsanity.com/paper/2502.07785/) (2025)
- [VOODOO XP: Expressive One-Shot Head Reenactment for VR Telepresence](https://mlsanity.com/paper/2405.16204/) (2024)
- [Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversation](https://mlsanity.com/paper/2601.00664/) (2026)
- [Joker: Conditional 3D Head Synthesis with Extreme Facial Expressions](https://mlsanity.com/paper/2410.16395/) (2024)
- [Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generation](https://mlsanity.com/paper/2504.02542/) (2025)
- [EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion](https://mlsanity.com/paper/2411.16726/) (2024)
- [SurFhead: Affine Rig Blending for Geometrically Accurate 2D Gaussian Surfel Head Avatars](https://mlsanity.com/paper/2410.11682/) (2024)
- [FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Cases](https://mlsanity.com/paper/2507.01390/) (2025)
- [MegActor-Sigma: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformer](https://mlsanity.com/paper/2408.14975/) (2024)
- [KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation](https://mlsanity.com/paper/2503.01715/) (2025)
- [TalkingHeadBench: A Multi-Modal Benchmark & Analysis of Talking-Head DeepFake Detection](https://mlsanity.com/paper/2505.24866/) (2025)
- [DH-FaceVid-1K: A Large-Scale High-Quality Dataset for Face Video Generation](https://mlsanity.com/paper/2410.07151/) (2024)
- [JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation](https://mlsanity.com/paper/2411.09209/) (2024)
- [AV-Flow: Transforming Text to Audio-Visual Human-like Interactions](https://mlsanity.com/paper/2502.13133/) (2025)
- [OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking](https://mlsanity.com/paper/2504.02433/) (2025)
- [GHOST 2.0: generative high-fidelity one shot transfer of heads](https://mlsanity.com/paper/2502.18417/) (2025)
- [DyStream: Streaming Dyadic Talking Heads Generation via Flow Matching-based Autoregressive Model](https://mlsanity.com/paper/2512.24408/) (2025)
- [EmojiDiff: Advanced Facial Expression Control with High Identity Preservation in Portrait Generation](https://mlsanity.com/paper/2412.01254/) (2024)
- [FaceCrafter: Identity-Conditional Diffusion with Disentangled Control over Facial Pose, Expression, and Emotion](https://mlsanity.com/paper/2505.15313/) (2025)
- [DeX-Portrait: Disentangled and Expressive Portrait Animation via Explicit and Latent Motion Representations](https://mlsanity.com/paper/2512.15524/) (2025)
- [Controlling Avatar Diffusion with Learnable Gaussian Embedding](https://mlsanity.com/paper/2503.15809/) (2025)
- [MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation](https://mlsanity.com/paper/2506.22065/) (2025)
- [JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching](https://mlsanity.com/paper/2506.23552/) (2025)

## Abstract (arXiv)

Head avatars animated by visual signals have gained popularity, particularly in cross-driving synthesis where the driver differs from the animated character, a challenging but highly practical approach. The recently presented MegaPortraits model has demonstrated state-of-the-art results in this domain. We conduct a deep examination and evaluation of this model, with a particular focus on its latent space for facial expression descriptors, and uncover several limitations with its ability to express intense face motions. To address these limitations, we propose substantial changes in both training pipeline and model architecture, to introduce our EMOPortraits model, where we: Enhance the model's capability to faithfully support intense, asymmetric face expressions, setting a new state-of-the-art result in the emotion transfer task, surpassing previous methods in both metrics and quality. Incorporate speech-driven mode to our model, achieving top-tier performance in audio-driven facial animation, making it possible to drive source identity through diverse modalities, including visual signal, audio, or a blend of both. We propose a novel multi-view video dataset featuring a wide range of intense and asymmetric facial expressions, filling the gap with absence of such data in existing datasets.