mlsanityData to 14 Oct 2026

OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation

arXiv 2506.18866 ↗First posted 23 Jun 2025cs.CV cs.AI cs.MMcited by 83

Not on the map. Why?

Abstract (arXiv)

Significant progress has been made in audio-driven human animation, while most existing methods focus mainly on facial movements, limiting their ability to create full-body animations with natural synchronization and fluidity. They also struggle with precise prompt control for fine-grained generation. To tackle these challenges, we introduce OmniAvatar, an innovative audio-driven full-body video generation model that enhances human animation with improved lip-sync accuracy and natural movements. OmniAvatar introduces a pixel-wise multi-hierarchical audio embedding strategy to better capture audio features in the latent space, enhancing lip-syncing across diverse scenes. To preserve the capability for prompt-driven control of foundation models while effectively incorporating audio features, we employ a LoRA-based training approach. Extensive experiments show that OmniAvatar surpasses existing models in both facial and semi-body video generation, offering precise text-based control for creating videos in various domains, such as podcasts, human interactions, dynamic scenes, and singing. Our project page is https://omni-avatar.github.io/.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

30 of 39 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  3. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  6. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  8. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  9. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 503
  10. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  11. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 206
  12. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 202
18 more
  1. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  2. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 194
  3. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 162
  4. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
  5. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 129
  6. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 122
  7. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  8. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 97
  9. 2025FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesiscited by 77
  10. 2024EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animationcited by 72
  11. 2024Sonic: Shifting Focus to Global Audio Perception in Portrait Animationcited by 72
  12. 2025HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characterscited by 69
  13. 2025Goku: Flow Based Video Generative Foundation Modelscited by 67
  14. 2025Let Them Talk: Audio-Driven Multi-Person Conversational Video Generationcited by 66
  15. 2024CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attentioncited by 39
  16. 2025EMO2: End-Effector Guided Audio-Driven Avatar Video Generationcited by 33
  17. 2024TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolationcited by 32
  18. 2025MoCha: Towards Movie-Grade Talking Character Synthesiscited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 30; 2026: 53.

83 papers.

Sort:

By area:

  1. 2025StableAvatar: Infinite-Length Audio-Driven Avatar Video Generationcited by 44
  2. 2025InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbingcited by 41
  3. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 38
  4. 2025OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulationcited by 32
  5. 2025EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animationcited by 29
  6. 2025StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatarscited by 25
  7. 2025Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesiscited by 21
  8. 2025AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinementcited by 20
  9. 2025KlingAvatar 2.0 Technical Reportcited by 13
  10. 2026Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillationcited by 13
  11. 2026FlowAct-R1: Towards Interactive Humanoid Video Generationcited by 11
  12. 2026TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captionscited by 11
38 more
  1. 2026VideoVeritas: AI-Generated Video Detection via Perception Pretext Reinforcement Learningcited by 11
  2. 2025SoulX-FlashTalk: Real-Time Infinite Streaming of Audio-Driven Avatars via Self-Correcting Bidirectional Distillationcited by 10
  3. 2025InfinityHuman: Towards Long-Term Audio-Driven Humancited by 9
  4. 2025LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillationcited by 9
  5. 2025Playmate2: Training-Free Multi-Character Audio-Driven Animation via Diffusion Transformer with Reward Feedbackcited by 8
  6. 2026OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generationcited by 8
  7. 2026AvatarForcing: One-Step Streaming Talking Avatars via Local-Future Sliding-Window Denoisingcited by 6
  8. 2026LongCat-Video-Avatar 1.5 Technical Reportcited by 6
  9. 2025ActAvatar: Temporally-Aware Precise Action Control for Talking Avatarscited by 5
  10. 2026InteractiveAvatar: Real-Time Streaming Video Generation for Consistent and Intent-Aware Avatarscited by 5
  11. 2025Lookahead Anchoring: Preserving Character Identity in Audio-Driven Human Animationcited by 4
  12. 2025EvalTalker: Learning to Evaluate Real-Portrait-Driven Multi-Subject Talking Humanscited by 4
  13. 2026Improving Joint Audio-Video Generation with Cross-Modal Context Learningcited by 4
  14. 2026Baton: Explicit Semantic Blueprints for Joint Video-Audio Generationcited by 4
  15. 2026Vidu S1: A Real-Time Interactive Video Generation Modelcited by 4
  16. 2025JAM-Flow: Joint Audio-Motion Synthesis with Flow Matchingcited by 3
  17. 2025FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for Audio-Driven Portrait Animationcited by 3
  18. 2025THEval. Evaluation Framework for Talking Head Video Generationcited by 3
  19. 2025JoyStreamer-Flash: Real-time and Infinite Audio-Driven Avatar Generation with Autoregressive Diffusioncited by 3
  20. 2025Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animationcited by 3
  21. 2025TalkVerse: Democratizing Minute-Long Audio-Driven Video Generationcited by 3
  22. 2025Active Intelligence in Video Avatars via Closed-loop World Modelingcited by 3
  23. 2026Making Avatars Interact: Towards Text-Driven Human-Object Interaction for Controllable Talking Avatarscited by 3
  24. 2026Gloria: Consistent Character Video Generation via Content Anchorscited by 3
  25. 2026OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generationcited by 3
  26. 2026CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generationcited by 3
  27. 2026Image-to-Video Diffusion: From Foundations to Open Frontierscited by 3
  28. 2025McSc: Motion-Corrective Preference Alignment for Video Generation with Self-Critic Hierarchical Reasoningcited by 2
  29. 2025Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Lengthcited by 2
  30. 2026EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformerscited by 2
  31. 2026SoulX-LiveAct: Towards Hour-Scale Real-Time Human Animation with Neighbor Forcing and ConvKV Memorycited by 2
  32. 2026TMD-Bench: A Multi-Level Evaluation Paradigm for Music-Dance Co-Generationcited by 2
  33. 2026Audio-Visual Intelligence in Large Foundation Modelscited by 2
  34. 2026Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronizationcited by 2
  35. 2026InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videoscited by 2
  36. 2025FluentAvatar: Flicker-Free Talking-Head Animation via Phoneme-Guided Autoregressive Modelingcited by 1
  37. 2025SyncLipMAE: Contrastive Masked Pretraining for Audio-Visual Talking-Face Representationcited by 1
  38. 2025VividAnimator: An End-to-End Audio and Pose-driven Half-Body Human Animation Frameworkcited by 1

50 most cited of 83 All 83 (JSON)