mlsanityData to 7 Oct 2026

CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attention

arXiv 2409.01876 ↗First posted 3 Sep 2024cs.CV cs.AIcited by 38

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Diffusion-based video generation technology has advanced significantly, catalyzing a proliferation of research in human animation. However, the majority of these studies are confined to same-modality driving settings, with cross-modality human body animation remaining relatively underexplored. In this paper, we introduce, an end-to-end audio-driven human animation framework that ensures hand integrity, identity consistency, and natural motion. The key design of CyberHost is the Region Codebook Attention mechanism, which improves the generation quality of facial and hand animations by integrating fine-grained local features with learned motion pattern priors. Furthermore, we have developed a suite of human-prior-guided training strategies, including body movement map, hand clarity score, pose-aligned reference feature, and local enhancement supervision, to improve synthesis results. To our knowledge, CyberHost is the first end-to-end audio-driven human diffusion model capable of facilitating zero-shot video generation within the scope of human body. Extensive experiments demonstrate that CyberHost surpasses previous works in both quantitative and qualitative aspects.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 42 of its 55 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

42 of 55 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Make-Your-Anchor: A Diffusion-based 2D Avatar Generation Framework25 Mar
  2. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis13 Mar
  3. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions27 Feb
  4. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation16 Mar
  5. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation6 Jan

Other references · 37

  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  5. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  6. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  8. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  9. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  10. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 573
  11. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  12. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
25 more
  1. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  2. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  3. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  4. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  5. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  6. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 307
  7. 2021Motion Representations for Articulated Animationcited by 253
  8. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  9. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  10. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 196
  11. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  12. 2019Learning Individual Styles of Conversational Gesturecited by 190
  13. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  14. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  15. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  16. 2024Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesiscited by 153
  17. 2023TryOnDiffusion: A Tale of Two UNetscited by 141
  18. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 141
  19. 2024V-Express: Conditional Dropout for Progressive Training of Portrait Video Generationcited by 85
  20. 2023DR2: Diffusion-based Robust Degradation Remover for Blind Face Restorationcited by 79
  21. 2024EasyAnimate: High-Performance Video Generation Framework with Hybrid Windows Attention and Reward Backpropagationcited by 76
  22. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 57
  23. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 54
  24. 2023DreamVideo: High-Fidelity Image-to-Video Generation with Image Retention and Text Guidancecited by 20
  25. 2024MobilePortrait: Real-Time One-Shot Neural Head Avatars on Mobile Devicescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 29; 2026: 4.

38 papers.

Sort:

By area:

  1. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  2. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  3. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 95
  4. 2025OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animationcited by 77
  5. 2024EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animationcited by 69
  6. 2025HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characterscited by 64
  7. 2025HumanDiT: Pose-Guided Diffusion Transformer for Long-form Human Motion Video Generationcited by 38
  8. 2024The Dawn of Video Generation: Preliminary Explorations with SORA-like Modelscited by 30
  9. 2025EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animationcited by 28
  10. 2025StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatarscited by 24
  11. 2025InterActHuman: Multi-Concept Human Animation with Layout-Aligned Audio Conditionscited by 20
  12. 2025Hallo4: High-Fidelity Dynamic Portrait Animation via Direct Preference Optimizationcited by 16
26 more
  1. 2025FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Casescited by 14
  2. 2025HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animationcited by 13
  3. 2025From Efficiency to Adaptivity: A Deeper Look at Adaptive Reasoning in Large Language Modelscited by 11
  4. 2025AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformerscited by 10
  5. 2025InfinityHuman: Towards Long-Term Audio-Driven Humancited by 9
  6. 2025ChatAnyone: Stylized Real-time Portrait Video Generation with Hierarchical Motion Diffusion Modelcited by 8
  7. 2025LLIA -- Enabling Low-Latency Interactive Avatars: Real-Time Audio-Driven Portrait Video Generation with Diffusion Modelscited by 7
  8. 2025Playmate2: Training-Free Multi-Character Audio-Driven Animation via Diffusion Transformer with Reward Feedbackcited by 7
  9. 2025Video Motion Graphscited by 5
  10. 2025EDTalk++: Full Disentanglement for Controllable Talking Head Synthesiscited by 5
  11. 2024FADA: Fast Diffusion Avatar Synthesis with Mixed-Supervised Multi-CFG Distillationcited by 4
  12. 2025InterAnimate: Taming Region-aware Diffusion Model for Realistic Human Interaction Animationcited by 4
  13. 2025Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformerscited by 3
  14. 2026Making Avatars Interact: Towards Text-Driven Human-Object Interaction for Controllable Talking Avatarscited by 3
  15. 2026Gloria: Consistent Character Video Generation via Content Anchorscited by 3
  16. 2025MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animationcited by 2
  17. 2025TalkingPose: Efficient Face and Gesture Animation with Feedback-guided Diffusion Modelcited by 2
  18. 2026CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generationcited by 2
  19. 2025Joint Learning of Depth and Appearance for Portrait Image Animationcited by 1
  20. 2025MMGT: Motion Mask Guided Two-Stage Network for Co-Speech Gesture Video Generationcited by 1
  21. 2025Democratizing High-Fidelity Co-Speech Gesture Video Generationcited by 1
  22. 2025Input-Aware Sparse Attention for Real-Time Co-Speech Video Generationcited by 1
  23. 2025TalkVerse: Democratizing Minute-Long Audio-Driven Video Generationcited by 1
  24. 2025EasyGenNet: An Efficient Framework for Audio-Driven Gesture Video Generation Based on Diffusion Modelcited by 0
  25. 2025A Survey of Body and Face Motion: Datasets, Performance Evaluation Metrics and Generative Techniquescited by 0
  26. 2026EchoTorrent: Towards Swift, Sustained, and Streaming Multi-Modal Video Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Unit Enhancement and Guidance Framework for Audio-Driven Avatar Video Generation
  2. 2025EMO2: End-Effector Guided Audio-Driven Avatar Video Generation
  3. 2023Dancing Avatar: Pose and Text-Guided Human Motion Videos Synthesis with Image Diffusion Model
  4. 2025FLAP: Fully-controllable Audio-driven Portrait Video Generation through 3D head conditioned diffusion model
  5. 2025MOSPA: Human Motion Generation Driven by Spatial Audio
  6. 2026Mamba-Enhanced Implicit Motion Learning for Audio-Driven Portrait Animation
  7. 2024AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation
  8. 2024Real-time One-Step Diffusion-based Expressive Portrait Videos Generation