mlsanityData to 7 Oct 2026

EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions

arXiv 2402.17485 ↗First posted 27 Feb 2024cs.CVcited by 233

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In this work, we tackle the challenge of enhancing the realism and expressiveness in talking head video generation by focusing on the dynamic and nuanced relationship between audio cues and facial movements. We identify the limitations of traditional techniques that often fail to capture the full spectrum of human expressions and the uniqueness of individual facial styles. To address these issues, we propose EMO, a novel framework that utilizes a direct audio-to-video synthesis approach, bypassing the need for intermediate 3D models or facial landmarks. Our method ensures seamless frame transitions and consistent identity preservation throughout the video, resulting in highly expressive and lifelike animations. Experimental results demonsrate that EMO is able to produce not only convincing speaking videos but also singing videos in various styles, significantly outperforming existing state-of-the-art methodologies in terms of expressiveness and realism.

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 34 of its 45 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

34 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronization18 Aug
  2. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation10 Jan
  3. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation6 Jan

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  12. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
19 more
  1. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  2. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  3. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  4. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  5. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  6. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  7. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  8. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  9. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  10. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  11. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 293
  12. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  13. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  14. 2022VFHQ: A High-Quality Dataset and Benchmark for Video Face Super-Resolutioncited by 174
  15. 2023TryOnDiffusion: A Tale of Two UNetscited by 141
  16. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  17. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  18. 2023VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Priorcited by 47
  19. 2023MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentionscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 63; 2025: 125; 2026: 45.

233 papers.

Sort:

By area:

  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  2. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 202
  3. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  4. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 191
  5. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  6. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 174
  7. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 172
  8. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  9. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 121
  10. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 115
  11. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  12. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 95
38 more
  1. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 85
  2. 2024V-Express: Conditional Dropout for Progressive Training of Portrait Video Generationcited by 85
  3. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 84
  4. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  5. 2024Sonic: Shifting Focus to Global Audio Perception in Portrait Animationcited by 70
  6. 2024EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animationcited by 69
  7. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  8. 2024Progressive Autoregressive Video Diffusion Modelscited by 65
  9. 2025Let Them Talk: Audio-Driven Multi-Person Conversational Video Generationcited by 64
  10. 2025Animate Anyone 2: High-Fidelity Character Image Animation with Environment Affordancecited by 60
  11. 2025Wan-S2V: Audio-Driven Cinematic Video Generationcited by 60
  12. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  13. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 54
  14. 2024LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervisioncited by 53
  15. 2024CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Modelscited by 52
  16. 2025Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Modelcited by 49
  17. 2025LAM: Large Avatar Model for One-shot Animatable Gaussian Headcited by 43
  18. 2024Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesiscited by 41
  19. 2025StableAvatar: Infinite-Length Audio-Driven Avatar Video Generationcited by 41
  20. 2025Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animationcited by 41
  21. 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portraitcited by 40
  22. 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
  23. 2025InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbingcited by 40
  24. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  25. 2024LeviTor: 3D Trajectory Oriented Image-to-Video Synthesiscited by 39
  26. 2024CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attentioncited by 38
  27. 2024MEMO: Memory-Guided Diffusion for Expressive Talking Video Generationcited by 38
  28. 2025DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidancecited by 36
  29. 2024FaithDiff: Unleashing Diffusion Priors for Faithful Image Super-resolutioncited by 31
  30. 2025MoCha: Towards Movie-Grade Talking Character Synthesiscited by 31
  31. 2024The Dawn of Video Generation: Preliminary Explorations with SORA-like Modelscited by 30
  32. 2025OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulationcited by 30
  33. 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 29
  34. 2025Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Datasetcited by 28
  35. 2025EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animationcited by 28
  36. 2024MegActor: Harness the Power of Raw Video for Vivid Portrait Animationcited by 27
  37. 2024MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Samplingcited by 27
  38. 2025SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generationcited by 27

50 most cited of 233 All 233 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024EMOdiffhead: Continuously Emotional Control in Talking Head Generation via Diffusion
  2. 2025X-Actor: Emotional and Expressive Long-Range Portrait Acting from Audio
  3. 2025EMO2: End-Effector Guided Audio-Driven Avatar Video Generation
  4. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation
  5. 2025FLAP: Fully-controllable Audio-driven Portrait Video Generation through 3D head conditioned diffusion model
  6. 2025EmoDiffusion: Enhancing Emotional 3D Facial Animation with Latent Diffusion Models
  7. 2025FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis
  8. 2024DreamHead: Learning Spatial-Temporal Correspondence via Hierarchical Diffusion for Audio-driven Talking Head Synthesis