# Emotion-Controllable Generalized Talking Face Generation

arXiv: https://arxiv.org/abs/2205.01155 · HTML: https://mlsanity.com/paper/2205.01155/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 2 May 2022.

## Builds on (most cited first, 22 of 22)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019)
- [Neural Voice Puppetry: Audio-driven Facial Reenactment](https://mlsanity.com/paper/1912.05566/) (2019)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [Talking Face Generation by Adversarially Disentangled Audio-Visual Representation](https://mlsanity.com/paper/1807.07860/) (2018)
- [Audio-Driven Emotional Video Portraits](https://mlsanity.com/paper/2104.07452/) (2021)
- [Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion](https://mlsanity.com/paper/2107.09293/) (2021)
- [Realistic Speech-Driven Facial Animation with GANs](https://mlsanity.com/paper/1906.06337/) (2019)
- [Talking-head Generation with Rhythmic Head Motion](https://mlsanity.com/paper/2007.08547/) (2020)
- [You said that?](https://mlsanity.com/paper/1705.02966/) (2017)
- [Everybody's Talkin': Let Me Talk as You Want](https://mlsanity.com/paper/2001.05201/) (2020)
- [Speech Driven Talking Face Generation from a Single Image and an Emotion Condition](https://mlsanity.com/paper/2008.03592/) (2020)
- [Identity-Preserving Realistic Talking Face Generation](https://mlsanity.com/paper/2005.12318/) (2020)

## Built on by (most cited first, 30 of 43)

- [StyleTalk: One-shot Talking Head Generation with Controllable Speaking Styles](https://mlsanity.com/paper/2301.01081/) (2023)
- [Efficient Emotional Adaptation for Audio-Driven Talking-Head Generation](https://mlsanity.com/paper/2309.04946/) (2023)
- [EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis](https://mlsanity.com/paper/2404.01647/) (2024)
- [FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization](https://mlsanity.com/paper/2403.06375/) (2024)
- [High-fidelity Generalized Emotional Talking Face Generation with Multi-modal Emotion Space Learning](https://mlsanity.com/paper/2305.02572/) (2023)
- [Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style](https://mlsanity.com/paper/2403.06365/) (2024)
- [TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles](https://mlsanity.com/paper/2304.00334/) (2023)
- [Say Anything with Any Style](https://mlsanity.com/paper/2403.06363/) (2024)
- [DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generation](https://mlsanity.com/paper/2312.13578/) (2023)
- [Speech-Driven 3D Face Animation with Composite and Regional Facial Movements](https://mlsanity.com/paper/2308.05428/) (2023)
- [Emotional Listener Portrait: Neural Listener Head Generation with Emotion](https://mlsanity.com/paper/2310.00068/) (2023)
- [MFR-Net: Multi-faceted Responsive Listening Head Generation via Denoising Diffusion Model](https://mlsanity.com/paper/2308.16635/) (2023)
- [DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Models](https://mlsanity.com/paper/2312.09767/) (2023)
- [EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion](https://mlsanity.com/paper/2411.16726/) (2024)
- [Emotionally Enhanced Talking Face Generation](https://mlsanity.com/paper/2303.11548/) (2023)
- [ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignment](https://mlsanity.com/paper/2308.14448/) (2023)
- [EmoTalker: Emotionally Editable Talking Face Generation via Diffusion Model](https://mlsanity.com/paper/2401.08049/) (2024)
- [StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads](https://mlsanity.com/paper/2409.09292/) (2024)
- [Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generation](https://mlsanity.com/paper/2504.18087/) (2025)
- [AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation](https://mlsanity.com/paper/2402.16124/) (2024)
- [OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking](https://mlsanity.com/paper/2504.02433/) (2025)
- [MMFace4D: A Large-Scale Multi-Modal 4D Face Dataset for Audio-Driven 3D Face Animation](https://mlsanity.com/paper/2303.09797/) (2023)
- [ConsistentAvatar: Learning to Diffuse Fully Consistent Talking Head Avatar with Temporal Guidance](https://mlsanity.com/paper/2411.15436/) (2024)
- [EMOdiffhead: Continuously Emotional Control in Talking Head Generation via Diffusion](https://mlsanity.com/paper/2409.07255/) (2024)
- [EDTalk++: Full Disentanglement for Controllable Talking Head Synthesis](https://mlsanity.com/paper/2508.13442/) (2025)
- [Facial Expression Video Generation Based-On Spatio-temporal Convolutional GAN: FEV-GAN](https://mlsanity.com/paper/2210.11182/) (2022)
- [EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation](https://mlsanity.com/paper/2402.01422/) (2024)
- [JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation](https://mlsanity.com/paper/2409.12156/) (2024)
- [Learning Frame-Wise Emotion Intensity for Audio-Driven Talking-Head Generation](https://mlsanity.com/paper/2409.19501/) (2024)
- [GMTalker: Gaussian Mixture-based Audio-Driven Emotional Talking Video Portraits](https://mlsanity.com/paper/2312.07669/) (2023)

## Abstract (arXiv)

Despite the significant progress in recent years, very few of the AI-based talking face generation methods attempt to render natural emotions. Moreover, the scope of the methods is majorly limited to the characteristics of the training dataset, hence they fail to generalize to arbitrary unseen faces. In this paper, we propose a one-shot facial geometry-aware emotional talking face generation method that can generalize to arbitrary faces. We propose a graph convolutional neural network that uses speech content feature, along with an independent emotion input to generate emotion and speech-induced motion on facial geometry-aware landmark representation. This representation is further used in our optical flow-guided texture generation network for producing the texture. We propose a two-branch texture generation network, with motion and texture branches designed to consider the motion and texture content independently. Compared to the previous emotion talking face methods, our method can adapt to arbitrary faces captured in-the-wild by fine-tuning with only a single image of the target identity in neutral emotion.