# End-to-End Speech-Driven Facial Animation with Temporal GANs

arXiv: https://arxiv.org/abs/1805.09313 · HTML: https://mlsanity.com/paper/1805.09313/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2018.

## Built on by (most cited first, 30 of 31)

- [Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer](https://mlsanity.com/paper/2204.03638/) (2022)
- [PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering](https://mlsanity.com/paper/2109.08379/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model](https://mlsanity.com/paper/2205.15278/) (2022)
- [CCVS: Context-aware Controllable Video Synthesis](https://mlsanity.com/paper/2107.08037/) (2021)
- [Synthetic Data -- what, why and how?](https://mlsanity.com/paper/2205.03257/) (2022)
- [FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models](https://mlsanity.com/paper/2312.08459/) (2023)
- [SPACE: Speech-driven Portrait Animation with Controllable Expression](https://mlsanity.com/paper/2211.09809/) (2022)
- [Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronization](https://mlsanity.com/paper/2308.09716/) (2023)
- [Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation](https://mlsanity.com/paper/2109.10595/) (2021)
- [Face Generation and Editing with StyleGAN: A Survey](https://mlsanity.com/paper/2212.09102/) (2022)
- [Speech Driven Video Editing via an Audio-Conditioned Diffusion Model](https://mlsanity.com/paper/2301.04474/) (2023)
- [Audio-Driven Talking Face Video Generation with Dynamic Convolution Kernels](https://mlsanity.com/paper/2201.05986/) (2022)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision](https://mlsanity.com/paper/2303.17200/) (2023)
- [AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person](https://mlsanity.com/paper/2108.04325/) (2021)
- [Deep Person Generation: A Survey from the Perspective of Face, Pose and Cloth Synthesis](https://mlsanity.com/paper/2109.02081/) (2021)
- [Talking Head from Speech Audio using a Pre-trained Image Generator](https://mlsanity.com/paper/2209.04252/) (2022)
- [Multi-modality Deep Restoration of Extremely Compressed Face Videos](https://mlsanity.com/paper/2107.05548/) (2021)
- [Talking Head Generation with Audio and Speech Related Facial Action Units](https://mlsanity.com/paper/2110.09951/) (2021)
- [KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolution](https://mlsanity.com/paper/2505.00497/) (2025)
- [VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selection](https://mlsanity.com/paper/2206.07458/) (2022)
- [READ Avatars: Realistic Emotion-controllable Audio Driven Avatars](https://mlsanity.com/paper/2303.00744/) (2023)
- [Emotional Conversation: Empowering Talking Faces with Cohesive Expression, Gaze and Pose Generation](https://mlsanity.com/paper/2406.07895/) (2024)
- [Laughing Matters: Introducing Laughing-Face Generation using Diffusion Models](https://mlsanity.com/paper/2305.08854/) (2023)
- [Speech-Gesture GAN: Gesture Generation for Robots and Embodied Agents](https://mlsanity.com/paper/2309.09346/) (2023)
- [One Shot Audio to Animated Video Generation](https://mlsanity.com/paper/2102.09737/) (2021)
- [Motion and Context-Aware Audio-Visual Conditioned Video Prediction](https://mlsanity.com/paper/2212.04679/) (2022)
- [IFaceUV: Intuitive Motion Facial Image Generation by Identity Preservation via UV map](https://mlsanity.com/paper/2306.04957/) (2023)
- [FlashLips: 100-FPS Mask-Free Latent Lip-Sync using Reconstruction Instead of Diffusion or GANs](https://mlsanity.com/paper/2512.20033/) (2025)

## Abstract (arXiv)

Speech-driven facial animation is the process which uses speech signals to automatically synthesize a talking character. The majority of work in this domain creates a mapping from audio features to visual features. This often requires post-processing using computer graphics techniques to produce realistic albeit subject dependent results. We present a system for generating videos of a talking head, using a still image of a person and an audio clip containing speech, that doesn't rely on any handcrafted intermediate features. To the best of our knowledge, this is the first method capable of generating subject independent realistic videos directly from raw audio. Our method can generate videos which have (a) lip movements that are in sync with the audio and (b) natural facial expressions such as blinks and eyebrow movements. We achieve this by using a temporal GAN with 2 discriminators, which are capable of capturing different aspects of the video. The effect of each component in our system is quantified through an ablation study. The generated videos are evaluated based on their sharpness, reconstruction quality, and lip-reading accuracy. Finally, a user study is conducted, confirming that temporal GANs lead to more natural sequences than a static GAN-based approach.