mlsanityData to 7 Oct 2026

AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation

arXiv 2403.17694 ↗First posted 26 Mar 2024cs.CV cs.GR eess.IVcited by 191

Not on the map. Why?

Cited by 65 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this study, we propose AniPortrait, a novel framework for generating high-quality animation driven by audio and a reference portrait image. Our methodology is divided into two stages. Initially, we extract 3D intermediate representations from audio and project them into a sequence of 2D facial landmarks. Subsequently, we employ a robust diffusion model, coupled with a motion module, to convert the landmark sequence into photorealistic and temporally consistent portrait animation. Experimental results demonstrate the superiority of AniPortrait in terms of facial naturalness, pose diversity, and visual quality, thereby offering an enhanced perceptual experience. Moreover, our methodology exhibits considerable potential in terms of flexibility and controllability, which can be effectively applied in areas such as facial motion editing or face reenactment. We release code and model weights at https://github.com/scutzzj/AniPortrait

Key facts. Cited by 191 papers in the map (data through 7 Oct 2026). Builds on 17 of its 19 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

17 of 19 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  8. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  9. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  10. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  11. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  12. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
5 more
  1. 2022VFHQ: A High-Quality Dataset and Benchmark for Video Face Super-Resolutioncited by 174
  2. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  3. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  4. 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 73
  5. 2023GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generationcited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 36; 2025: 111; 2026: 44.

191 papers.

Sort:

By area:

  1. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 202
  2. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  3. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  4. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  5. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 172
  6. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 121
  7. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 95
  8. 2025Controllable Video Generation: A Surveycited by 93
  9. 2024V-Express: Conditional Dropout for Progressive Training of Portrait Video Generationcited by 85
  10. 2025FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesiscited by 77
  11. 2025OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animationcited by 77
  12. 2025Wan-Animate: Unified Character Animation and Replacement with Holistic Replicationcited by 76
38 more
  1. 2024Sonic: Shifting Focus to Global Audio Perception in Portrait Animationcited by 70
  2. 2024EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animationcited by 69
  3. 2025Let Them Talk: Audio-Driven Multi-Person Conversational Video Generationcited by 64
  4. 2025HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animationcited by 55
  5. 2025Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Modelcited by 49
  6. 2024Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesiscited by 41
  7. 2025StableAvatar: Infinite-Length Audio-Driven Avatar Video Generationcited by 41
  8. 2025Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animationcited by 41
  9. 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portraitcited by 40
  10. 2025InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbingcited by 40
  11. 2024MEMO: Memory-Guided Diffusion for Expressive Talking Video Generationcited by 38
  12. 2025MoCha: Towards Movie-Grade Talking Character Synthesiscited by 31
  13. 2025OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulationcited by 30
  14. 2025EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animationcited by 28
  15. 2025OmniSync: Towards Universal Lip Synchronization via Diffusion Transformerscited by 26
  16. 2025X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attentioncited by 26
  17. 2025Human Motion Video Generation: A Surveycited by 25
  18. 2025AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinementcited by 19
  19. 2026DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generationcited by 19
  20. 2025X-Dyna: Expressive Dynamic Human Image Animationcited by 18
  21. 2025Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generationcited by 17
  22. 2025TalkVid: A Large-Scale Diversified Dataset for Audio-Driven Talking Head Synthesiscited by 17
  23. 2025MVP4D: Multi-View Portrait Video Diffusion for Animatable 4D Avatarscited by 17
  24. 2024EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusioncited by 16
  25. 2025Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generationcited by 16
  26. 2025Hallo4: High-Fidelity Dynamic Portrait Animation via Direct Preference Optimizationcited by 16
  27. 2025AvatarArtist: Open-Domain 4D Avatarizationcited by 15
  28. 2025MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animationcited by 14
  29. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  30. 2025Audio-Sync Video Generation with Multi-Stream Temporal Controlcited by 14
  31. 2025FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Casescited by 14
  32. 2024MegActor-Sigma: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformercited by 13
  33. 2025HumanDreamer-X: Photorealistic Single-image Human Avatars Reconstruction via Gaussian Restorationcited by 13
  34. 2025SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformerscited by 13
  35. 2025MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animationcited by 12
  36. 2025FaceShot: Bring Any Character into Lifecited by 12
  37. 2025Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generationcited by 11
  38. 2025KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolutioncited by 11

50 most cited of 191 All 191 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expression
  2. 2025MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation
  3. 20253DiFACE: Synthesizing and Editing Holistic 3D Facial Animation
  4. 2024DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation
  5. 2024X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention
  6. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency
  7. 2026FG-Portrait: 3D Flow Guided Editable Portrait Animation
  8. 2024A Comprehensive Taxonomy and Analysis of Talking Head Synthesis: Techniques for Portrait Generation, Driving Mechanisms, and Editing