mlsanityData to 7 Oct 2026

Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency

arXiv 2409.02634 ↗First posted 4 Sep 2024cs.CVcited by 102

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

With the introduction of diffusion-based video generation techniques, audio-conditioned human video generation has recently achieved significant breakthroughs in both the naturalness of motion and the synthesis of portrait details. Due to the limited control of audio signals in driving human motion, existing methods often add auxiliary spatial signals to stabilize movements, which may compromise the naturalness and freedom of motion. In this paper, we propose an end-to-end audio-only conditioned video diffusion model named Loopy. Specifically, we designed an inter- and intra-clip temporal module and an audio-to-latents module, enabling the model to leverage long-term motion information from the data to learn natural motion patterns and improving audio-portrait movement correlation. This method removes the need for manually specified spatial motion templates used in existing methods to constrain motion during inference. Extensive experiments show that Loopy outperforms recent audio-driven portrait diffusion models, delivering more lifelike and high-quality results across various scenarios.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 44 of its 55 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

44 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attention3 Sep
  2. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis13 Mar
  3. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions27 Feb
  4. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation6 Jan

Other references · 40

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  8. 2022Video Diffusion Modelscited by 2,280
  9. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  11. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  12. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
28 more
  1. 2020First Order Motion Model for Image Animationcited by 698
  2. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  3. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  4. 2023ModelScope Text-to-Video Technical Reportcited by 651
  5. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  6. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  7. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  8. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  9. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  10. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  11. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  12. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 307
  13. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  14. 2021Motion Representations for Articulated Animationcited by 253
  15. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  16. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 202
  17. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  18. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  19. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  20. 2017Video Generation From Textcited by 153
  21. 2023TryOnDiffusion: A Tale of Two UNetscited by 141
  22. 2022MegaPortraits: One-shot Megapixel Neural Head Avatarscited by 131
  23. 2022Generating Long Videos of Dynamic Scenescited by 113
  24. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  25. 2024V-Express: Conditional Dropout for Progressive Training of Portrait Video Generationcited by 85
  26. 2023GAIA: Zero-shot Talking Avatar Generationcited by 43
  27. 2023DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generationcited by 24
  28. 2024MobilePortrait: Real-Time One-Shot Neural Head Avatars on Mobile Devicescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 70; 2026: 20.

102 papers.

Sort:

By area:

  1. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  2. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 121
  3. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 95
  4. 2025HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generationcited by 84
  5. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 84
  6. 2025FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesiscited by 77
  7. 2025OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animationcited by 77
  8. 2024Sonic: Shifting Focus to Global Audio Perception in Portrait Animationcited by 70
  9. 2025HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characterscited by 64
  10. 2025Let Them Talk: Audio-Driven Multi-Person Conversational Video Generationcited by 64
  11. 2024LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervisioncited by 53
  12. 2025Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Modelcited by 49
38 more
  1. 2024Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesiscited by 41
  2. 2025StableAvatar: Infinite-Length Audio-Driven Avatar Video Generationcited by 41
  3. 2025Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animationcited by 41
  4. 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portraitcited by 40
  5. 2025InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbingcited by 40
  6. 2024MEMO: Memory-Guided Diffusion for Expressive Talking Video Generationcited by 38
  7. 2025VRAG: Learning World Models for Interactive Video Generationcited by 37
  8. 2025SkyReels-A1: Expressive Portrait Animation in Video Diffusion Transformerscited by 33
  9. 2025MoCha: Towards Movie-Grade Talking Character Synthesiscited by 31
  10. 2025OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulationcited by 30
  11. 2024MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Samplingcited by 27
  12. 2025OmniSync: Towards Universal Lip Synchronization via Diffusion Transformerscited by 26
  13. 2025FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generationcited by 21
  14. 2025InterActHuman: Multi-Concept Human Animation with Layout-Aligned Audio Conditionscited by 20
  15. 2025Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesiscited by 20
  16. 2025AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinementcited by 19
  17. 2025MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generationcited by 18
  18. 2025Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generationcited by 17
  19. 2026Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversationcited by 17
  20. 2024MimicTalk: Mimicking a personalized and expressive 3D talking face in minutescited by 16
  21. 2025MagicInfinite: Generating Infinite Talking Videos with Your Words and Voicecited by 16
  22. 2025Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generationcited by 16
  23. 2025Hallo4: High-Fidelity Dynamic Portrait Animation via Direct Preference Optimizationcited by 16
  24. 2025ARIG: Autoregressive Interactive Head Generation for Real-time Conversationscited by 16
  25. 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 16
  26. 2025X-Dancer: Expressive Music to Human Dance Video Generationcited by 15
  27. 2025Audio-Sync Video Generation with Multi-Stream Temporal Controlcited by 14
  28. 2025FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Casescited by 14
  29. 2025SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformerscited by 13
  30. 2026Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillationcited by 13
  31. 2025MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animationcited by 12
  32. 2025SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusioncited by 12
  33. 2025Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generationcited by 11
  34. 2025KlingAvatar 2.0 Technical Reportcited by 11
  35. 2024JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generationcited by 9
  36. 2025AV-Flow: Transforming Text to Audio-Visual Human-like Interactionscited by 9
  37. 2025InfinityHuman: Towards Long-Term Audio-Driven Humancited by 9
  38. 2025Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusioncited by 8

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Mamba-Enhanced Implicit Motion Learning for Audio-Driven Portrait Animation
  2. 2024Real-time One-Step Diffusion-based Expressive Portrait Videos Generation
  3. 2024DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation
  4. 2024LoopAnimate: Loopable Salient Object Animation
  5. 2025A Unit Enhancement and Guidance Framework for Audio-Driven Avatar Video Generation
  6. 2023Dancing Avatar: Pose and Text-Guided Human Motion Videos Synthesis with Image Diffusion Model
  7. 2025DEMO: Disentangled Motion Latent Flow Matching for Fine-Grained Controllable Talking Portrait Synthesis
  8. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models