# Emotionally Enhanced Talking Face Generation

arXiv: https://arxiv.org/abs/2303.11548 · HTML: https://mlsanity.com/paper/2303.11548/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 55 references that are in the map.
- First posted on arXiv 21 Mar 2023.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019)
- [Neural Voice Puppetry: Audio-driven Facial Reenactment](https://mlsanity.com/paper/1912.05566/) (2019)
- [Audio-Driven Emotional Video Portraits](https://mlsanity.com/paper/2104.07452/) (2021)
- [StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN](https://mlsanity.com/paper/2203.04036/) (2022)
- [Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion](https://mlsanity.com/paper/2107.09293/) (2021)
- [Realistic Speech-Driven Facial Animation with GANs](https://mlsanity.com/paper/1906.06337/) (2019)
- [Talking-head Generation with Rhythmic Head Motion](https://mlsanity.com/paper/2007.08547/) (2020)
- [FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning](https://mlsanity.com/paper/2108.07938/) (2021)
- [ReenactGAN: Learning to Reenact Faces via Boundary Transfer](https://mlsanity.com/paper/1807.11079/) (2018)
- [Everybody's Talkin': Let Me Talk as You Want](https://mlsanity.com/paper/2001.05201/) (2020)
- [Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward](https://mlsanity.com/paper/2103.00484/) (2021)
- [Speech Driven Talking Face Generation from a Single Image and an Emotion Condition](https://mlsanity.com/paper/2008.03592/) (2020)
- [Emotion-Controllable Generalized Talking Face Generation](https://mlsanity.com/paper/2205.01155/) (2022)
- [Product of Orthogonal Spheres Parameterization for Disentangled Representation Learning](https://mlsanity.com/paper/1907.09554/) (2019)
- [Attention-Based Lip Audio-Visual Synthesis for Talking Face Generation in the Wild](https://mlsanity.com/paper/2203.03984/) (2022)
- [Invertible Frowns: Video-to-Video Facial Emotion Translation](https://mlsanity.com/paper/2109.08061/) (2021)
- [DisCont: Self-Supervised Visual Attribute Disentanglement using Context Vectors](https://mlsanity.com/paper/2006.05895/) (2020)

## Built on by (most cited first, 14 of 14)

- [EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis](https://mlsanity.com/paper/2404.01647/) (2024)
- [FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization](https://mlsanity.com/paper/2403.06375/) (2024)
- [From Pixels to Portraits: A Comprehensive Survey of Talking Head Generation Techniques and Applications](https://mlsanity.com/paper/2308.16041/) (2023)
- [Design-o-meter: Towards Evaluating and Refining Graphic Designs](https://mlsanity.com/paper/2411.14959/) (2024)
- [PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation](https://mlsanity.com/paper/2412.07754/) (2024)
- [EDTalk++: Full Disentanglement for Controllable Talking Head Synthesis](https://mlsanity.com/paper/2508.13442/) (2025)
- [JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Model](https://mlsanity.com/paper/2408.01627/) (2024)
- [JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation](https://mlsanity.com/paper/2409.12156/) (2024)
- [FREAK: Frequency-modulated High-fidelity and Real-time Audio-driven Talking Portrait Synthesis](https://mlsanity.com/paper/2503.04067/) (2025)
- [Learning and Evaluating Human Preferences for Conversational Head Generation](https://mlsanity.com/paper/2307.10636/) (2023)
- [Cross-Modal Emotion Transfer for Emotion Editing in Talking Face Video](https://mlsanity.com/paper/2604.07786/) (2026)
- [G4G:A Generic Framework for High Fidelity Talking Face Generation with Fine-grained Intra-modal Alignment](https://mlsanity.com/paper/2402.18122/) (2024)
- [Embedded Representation Learning Network for Animating Styled Video Portrait](https://mlsanity.com/paper/2404.19038/) (2024)
- [Generative Learning as a Tool to Improve Perception of Emotional Body Motion Expressions](https://mlsanity.com/paper/2606.28769/) (2026)

## Abstract (arXiv)

Several works have developed end-to-end pipelines for generating lip-synced talking faces with various real-world applications, such as teaching and language translation in videos. However, these prior works fail to create realistic-looking videos since they focus little on people's expressions and emotions. Moreover, these methods' effectiveness largely depends on the faces in the training dataset, which means they may not perform well on unseen faces. To mitigate this, we build a talking face generation framework conditioned on a categorical emotion to generate videos with appropriate expressions, making them more realistic and convincing. With a broad range of six emotions, i.e., happiness, sadness, fear, anger, disgust, and neutral, we show that our model can adapt to arbitrary identities, emotions, and languages. Our proposed framework is equipped with a user-friendly web interface with a real-time experience for talking face generation with emotions. We also conduct a user study for subjective evaluation of our interface's usability, design, and functionality. Project page: https://midas.iiitd.edu.in/emo/