mlsanityData to 7 Oct 2026

VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time

arXiv 2404.10667 ↗First posted 16 Apr 2024cs.CVcited by 186

Not on the map. Why?

Cited by 53 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce VASA, a framework for generating lifelike talking faces with appealing visual affective skills (VAS) given a single static image and a speech audio clip. Our premiere model, VASA-1, is capable of not only generating lip movements that are exquisitely synchronized with the audio, but also producing a large spectrum of facial nuances and natural head motions that contribute to the perception of authenticity and liveliness. The core innovations include a holistic facial dynamics and head movement generation model that works in a face latent space, and the development of such an expressive and disentangled face latent space using videos. Through extensive experiments including evaluation on a set of new metrics, we show that our method significantly outperforms previous methods along various dimensions comprehensively. Our method not only delivers high video quality with realistic facial and head dynamics but also supports the online generation of 512x512 videos at up to 40 FPS with negligible starting latency. It paves the way for real-time engagements with lifelike avatars that emulate human conversational behaviors.

Key facts. Cited by 186 papers in the map (data through 7 Oct 2026). Builds on 57 of its 78 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

57 of 78 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  12. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
38 more
  1. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  2. 2020First Order Motion Model for Image Animationcited by 698
  3. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  4. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  5. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  6. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  7. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  8. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  9. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  10. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  11. 2016Generating Videos with Scene Dynamicscited by 376
  12. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  13. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  14. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  15. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  16. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  17. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  18. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 210
  19. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  20. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 202
  21. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 191
  22. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  23. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  24. 2018Lip Movements Generation at a Glancecited by 154
  25. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 148
  26. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generationcited by 142
  27. 2022MegaPortraits: One-shot Megapixel Neural Head Avatarscited by 131
  28. 2021Pixel Codec Avatarscited by 125
  29. 2020Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatarscited by 116
  30. 2020Neural Head Reenactment with Latent Pose Descriptorscited by 106
  31. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  32. 2022Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesiscited by 93
  33. 2023StyleTalk: One-shot Talking Head Generation with Controllable Speaking Stylescited by 92
  34. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  35. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  36. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 79
  37. 2022MetaPortrait: Identity-Preserving Talking Head Generation with Fast Personalized Adaptationcited by 69
  38. 2022L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environmentscited by 65

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 48; 2025: 96; 2026: 42.

186 papers.

Sort:

By area:

  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  2. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 202
  3. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  4. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 172
  5. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  6. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 121
  7. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  8. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 95
  9. 2025Controllable Video Generation: A Surveycited by 93
  10. 2024V-Express: Conditional Dropout for Progressive Training of Portrait Video Generationcited by 85
  11. 2024OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generationcited by 77
  12. 2025Wan-Animate: Unified Character Animation and Replacement with Holistic Replicationcited by 76
38 more
  1. 2024Sonic: Shifting Focus to Global Audio Perception in Portrait Animationcited by 70
  2. 2024EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animationcited by 69
  3. 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 61
  4. 2025HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animationcited by 55
  5. 2024CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Modelscited by 52
  6. 2024Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesiscited by 41
  7. 2025Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animationcited by 41
  8. 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portraitcited by 40
  9. 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
  10. 2024CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attentioncited by 38
  11. 2024MEMO: Memory-Guided Diffusion for Expressive Talking Video Generationcited by 38
  12. 2025DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidancecited by 36
  13. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  14. 2024The Dawn of Video Generation: Preliminary Explorations with SORA-like Modelscited by 30
  15. 2025OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulationcited by 30
  16. 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 29
  17. 2025EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animationcited by 28
  18. 2024MegActor: Harness the Power of Raw Video for Vivid Portrait Animationcited by 27
  19. 2024MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Samplingcited by 27
  20. 2025SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generationcited by 27
  21. 2025TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Modelscited by 26
  22. 2025X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attentioncited by 26
  23. 2025Human Motion Video Generation: A Surveycited by 25
  24. 2025StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatarscited by 24
  25. 2025Pippo: High-Resolution Multi-View Humans from a Single Imagecited by 20
  26. 2025InterActHuman: Multi-Concept Human Animation with Layout-Aligned Audio Conditionscited by 20
  27. 2025AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinementcited by 19
  28. 2025InsTaG: Learning Personalized 3D Talking Head from Few-Second Videocited by 18
  29. 2025MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generationcited by 18
  30. 2025Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generationcited by 17
  31. 2025TalkVid: A Large-Scale Diversified Dataset for Audio-Driven Talking Head Synthesiscited by 17
  32. 2025MVP4D: Multi-View Portrait Video Diffusion for Animatable 4D Avatarscited by 17
  33. 2026Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversationcited by 17
  34. 2024MimicTalk: Mimicking a personalized and expressive 3D talking face in minutescited by 16
  35. 2024EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusioncited by 16
  36. 2025Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generationcited by 16
  37. 2025Hallo4: High-Fidelity Dynamic Portrait Animation via Direct Preference Optimizationcited by 16
  38. 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 16

50 most cited of 186 All 186 (JSON)