# Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks

arXiv: https://arxiv.org/abs/1806.00154 · HTML: https://mlsanity.com/paper/1806.00154/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2018.

## Built on by (most cited first, 21 of 21)

- [EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model](https://mlsanity.com/paper/2205.15278/) (2022)
- [StyleTalk: One-shot Talking Head Generation with Controllable Speaking Styles](https://mlsanity.com/paper/2301.01081/) (2023)
- [DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering](https://mlsanity.com/paper/2201.00791/) (2022)
- [Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation](https://mlsanity.com/paper/2104.07995/) (2021)
- [TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles](https://mlsanity.com/paper/2304.00334/) (2023)
- [Speech Driven Video Editing via an Audio-Conditioned Diffusion Model](https://mlsanity.com/paper/2301.04474/) (2023)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [Deep Person Generation: A Survey from the Perspective of Face, Pose and Cloth Synthesis](https://mlsanity.com/paper/2109.02081/) (2021)
- [DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Models](https://mlsanity.com/paper/2312.09767/) (2023)
- [ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignment](https://mlsanity.com/paper/2308.14448/) (2023)
- [StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads](https://mlsanity.com/paper/2409.09292/) (2024)
- [AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation](https://mlsanity.com/paper/2402.16124/) (2024)
- [Emotional Talking Head Generation based on Memory-Sharing and Attention-Augmented Networks](https://mlsanity.com/paper/2306.03594/) (2023)
- [Text-to-Image Cross-Modal Generation: A Systematic Review](https://mlsanity.com/paper/2401.11631/) (2024)
- [EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation](https://mlsanity.com/paper/2402.01422/) (2024)
- [Controlled AutoEncoders to Generate Faces from Voices](https://mlsanity.com/paper/2107.07988/) (2021)
- [GMTalker: Gaussian Mixture-based Audio-Driven Emotional Talking Video Portraits](https://mlsanity.com/paper/2312.07669/) (2023)
- [A self-attention-based differentially private tabular GAN with high data utility](https://mlsanity.com/paper/2312.13031/) (2023)
- [Show Me Your Face, And I'll Tell You How You Speak](https://mlsanity.com/paper/2206.14009/) (2022)
- [Exploring Talking Head Models With Adjacent Frame Prior for Speech-Preserving Facial Expression Manipulation](https://mlsanity.com/paper/2601.12876/) (2026)
- [GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting](https://mlsanity.com/paper/2607.00959/) (2026)

## Abstract (arXiv)

Articulation, emotion, and personality play strong roles in the orofacial movements. To improve the naturalness and expressiveness of virtual agents (VAs), it is important that we carefully model the complex interplay between these factors. This paper proposes a conditional generative adversarial network, called conditional sequential GAN (CSG), which learns the relationship between emotion and lexical content in a principled manner. This model uses a set of articulatory and emotional features directly extracted from the speech signal as conditioning inputs, generating realistic movements. A key feature of the approach is that it is a speech-driven framework that does not require transcripts. Our experiments show the superiority of this model over three state-of-the-art baselines in terms of objective and subjective evaluations. When the target emotion is known, we propose to create emotionally dependent models by either adapting the base model with the target emotional data (CSG-Emo-Adapted), or adding emotional conditions as the input of the model (CSG-Emo-Aware). Objective evaluations of these models show improvements for the CSG-Emo-Adapted compared with the CSG model, as the trajectory sequences are closer to the original sequences. Subjective evaluations show significantly better results for this model compared with the CSG model when the target emotion is happiness.