mlsanityData to 7 Oct 2026

SPACE: Speech-driven Portrait Animation with Controllable Expression

arXiv 2211.09809 ↗First posted 17 Nov 2022cs.CVcited by 49

Not on the map. Why?

Abstract (arXiv)

Animating portraits using speech has received growing attention in recent years, with various creative and practical use cases. An ideal generated video should have good lip sync with the audio, natural facial expressions and head motions, and high frame quality. In this work, we present SPACE, which uses speech and a single image to generate high-resolution, and expressive videos with realistic head pose, without requiring a driving video. It uses a multi-stage approach, combining the controllability of facial landmarks with the high-quality synthesis power of a pretrained face generator. SPACE also allows for the control of emotions and their intensities. Our method outperforms prior methods in objective metrics for image quality and facial motions and is strongly preferred by users in pair-wise comparisons. The project website is available at https://deepimagination.cc/SPACE/

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

30 of 46 references.

References · 30

  1. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  2. 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 895
  3. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  4. 2020First Order Motion Model for Image Animationcited by 698
  5. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  6. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 631
  7. 2020Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstructioncited by 465
  8. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  9. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  10. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  11. 2021Motion Representations for Articulated Animationcited by 253
  12. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
18 more
  1. 2020Towards Fast, Accurate and Stable 3D Dense Face Alignmentcited by 219
  2. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  3. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  4. 2019Few-shot Video-to-Video Synthesiscited by 196
  5. 2021Audio-Driven Emotional Video Portraitscited by 180
  6. 2018GANimation: Anatomically-aware Facial Animation from a Single Imagecited by 160
  7. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  8. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 148
  9. 2018Face Alignment in Full Pose Range: A 3D Total Solutioncited by 147
  10. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  11. 2020Neural Head Reenactment with Latent Pose Descriptorscited by 106
  12. 2019MarioNETte: Few-shot Face Reenactment Preserving Identity of Unseen Targetscited by 90
  13. 2018Talking Face Generation by Conditional Recurrent Adversarial Networkcited by 84
  14. 2018VisemeNet: Audio-Driven Animator-Centric Speech Animationcited by 62
  15. 2022Implicit Warping for Animation with Image Setscited by 45
  16. 2018End-to-End Speech-Driven Facial Animation with Temporal GANscited by 31
  17. 2020Efficient Facial Feature Learning with Wide Ensemble-based Convolutional Neural Networkscited by 24
  18. 2019Wav2Pix: Speech-conditioned Face Generation using Generative Adversarial Networkscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 17; 2025: 12; 2026: 7.

49 papers.

Sort:

By area:

  1. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  2. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  3. 2024EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatarscited by 65
  4. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  5. 2023DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencodercited by 48
  6. 2023VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Priorcited by 47
  7. 2023GAIA: Zero-shot Talking Avatar Generationcited by 43
  8. 2023DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generationcited by 24
  9. 2023DaGAN++: Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 20
  10. 2025InsTaG: Learning Personalized 3D Talking Head from Few-Second Videocited by 18
  11. 2023Speech-Driven 3D Face Animation with Composite and Regional Facial Movementscited by 17
  12. 2025Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generationcited by 17
37 more
  1. 2024InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generationcited by 16
  2. 2024EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusioncited by 16
  3. 2023DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Modelscited by 13
  4. 2024GSTalker: Real-time Audio-Driven Talking Face Generation via Deformable Gaussian Splattingcited by 12
  5. 2024StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Headscited by 11
  6. 2024Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlookcited by 10
  7. 2025KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolationcited by 10
  8. 2025OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimickingcited by 9
  9. 2024AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generationcited by 8
  10. 2023FONT: Flow-guided One-shot Talking Head Generation with Natural Head Motionscited by 5
  11. 2024Emotional Conversation: Empowering Talking Faces with Cohesive Expression, Gaze and Pose Generationcited by 4
  12. 2025Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generationcited by 4
  13. 2023A Survey on Deep Multi-modal Learning for Body Language Recognition and Generationcited by 3
  14. 2024Dubbing for Everyone: Data-Efficient Visual Dubbing using Neural Rendering Priorscited by 3
  15. 2024JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Modelcited by 3
  16. 2024FD2Talk: Towards Generalized Talking Head Generation with Facial Decoupled Diffusion Modelcited by 3
  17. 2024DreamHead: Learning Spatial-Temporal Correspondence via Hierarchical Diffusion for Audio-driven Talking Head Synthesiscited by 3
  18. 2024Learning Frame-Wise Emotion Intensity for Audio-Driven Talking-Head Generationcited by 3
  19. 2023Laughing Matters: Introducing Laughing-Face Generation using Diffusion Modelscited by 2
  20. 2023GMTalker: Gaussian Mixture-based Audio-Driven Emotional Talking Video Portraitscited by 2
  21. 2024Controllable Talking Face Generation by Implicit Facial Keypoints Editingcited by 2
  22. 2025Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based Spatiotemporal Diffusion for Audio-driven Talking Portraitcited by 2
  23. 2026CogPortrait: Fine-Grained Eye-Region Control in Portrait Animation via Hierarchical Agent Planningcited by 2
  24. 2024Learning Online Scale Transformation for Talking Head Video Generationcited by 1
  25. 2024High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Modelcited by 1
  26. 2025Joint Learning of Depth and Appearance for Portrait Image Animationcited by 1
  27. 2025THEval. Evaluation Framework for Talking Head Video Generationcited by 1
  28. 2025IMTalker: Efficient Audio-driven Talking Face Generation with Implicit Motion Transfercited by 1
  29. 2026Cross-Modal Emotion Transfer for Emotion Editing in Talking Face Videocited by 1
  30. 2025StyGazeTalk: Learning Stylized Generation of Gaze and Head Dynamicscited by 0
  31. 2025AI killed the video star. Audio-driven diffusion model for expressive talking head generationcited by 0
  32. 20252K-Characters-10K-Stories: A Quality-Gated Stylized Narrative Dataset with Disentangled Control and Sequence Consistencycited by 0
  33. 2026Exploring Talking Head Models With Adjacent Frame Prior for Speech-Preserving Facial Expression Manipulationcited by 0
  34. 2026Real-Time Generation of Streamable Talking Portrait Video with Reference-Guided Deep Compression VAEscited by 0
  35. 2026ReFree: Towards Realistic Co-Speech Video Generation via Reward-Free RL and Multilevel Speech Guidancecited by 0
  36. 2026Proxy Avatar Meets Low-Rank Caching: Real-Time One-Shot Emotion-Controllable Portrait Animationcited by 0
  37. 2026Xemo-Talker: Unlock Emotions Explicitly for Audio-Driven Talking Portrait Synthesiscited by 0