mlsanityData to 7 Oct 2026

OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models

arXiv 2502.01061 ↗First posted 3 Feb 2025cs.CVcited by 127

Not on the map. Why?

Cited by 39 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

End-to-end human animation, such as audio-driven talking human generation, has undergone notable advancements in the recent few years. However, existing methods still struggle to scale up as large general video generation models, limiting their potential in real applications. In this paper, we propose OmniHuman, a Diffusion Transformer-based framework that scales up data by mixing motion-related conditions into the training phase. To this end, we introduce two training principles for these mixed conditions, along with the corresponding model architecture and inference strategy. These designs enable OmniHuman to fully leverage data-driven motion generation, ultimately achieving highly realistic human video generation. More importantly, OmniHuman supports various portrait contents (face close-up, portrait, half-body, full-body), supports both talking and singing, handles human-object interactions and challenging body poses, and accommodates different image styles. Compared to existing end-to-end audio-driven methods, OmniHuman not only produces more realistic videos, but also offers greater flexibility in inputs. It also supports multiple driving modalities (audio-driven, video-driven and combined driving signals). Video samples are provided on the ttfamily project page (https://omnihuman-lab.github.io)

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Builds on 73 of its 85 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

73 of 85 references.

References · 73

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2022Classifier-Free Diffusion Guidancecited by 5,717
  8. 2022Flow Matching for Generative Modelingcited by 5,426
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
38 more
  1. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  2. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  4. 2022Video Diffusion Modelscited by 2,280
  5. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  8. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  9. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  10. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  11. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  12. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  13. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  14. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  15. 2020First Order Motion Model for Image Animationcited by 698
  16. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  17. 2023ModelScope Text-to-Video Technical Reportcited by 651
  18. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  19. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  20. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  21. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  22. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  23. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  24. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  25. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  26. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  27. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  28. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  29. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  30. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  31. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  32. 2024Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidancecited by 271
  33. 2021Motion Representations for Articulated Animationcited by 253
  34. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  35. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  36. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  37. 2022Thin-Plate Spline Motion Model for Image Animationcited by 209
  38. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 83; 2026: 44.

127 papers.

Sort:

By area:

  1. 2025Controllable Video Generation: A Surveycited by 93
  2. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 84
  3. 2025Seaweed-7B: Cost-Effective Training of Video Generation Foundation Modelcited by 81
  4. 2025OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animationcited by 77
  5. 2025HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characterscited by 64
  6. 2025Let Them Talk: Audio-Driven Multi-Person Conversational Video Generationcited by 64
  7. 2025LHM: Large Animatable Human Reconstruction Model from a Single Image in Secondscited by 61
  8. 2025Wan-S2V: Audio-Driven Cinematic Video Generationcited by 60
  9. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 58
  10. 2026Helios: Real Real-Time Long Video Generation Modelcited by 56
  11. 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 45
  12. 2025HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioningcited by 44
38 more
  1. 2025StableAvatar: Infinite-Length Audio-Driven Avatar Video Generationcited by 41
  2. 2025InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbingcited by 40
  3. 2025DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidancecited by 36
  4. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 35
  5. 2025OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulationcited by 30
  6. 2025RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wildcited by 28
  7. 2025EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animationcited by 28
  8. 2025SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generationcited by 27
  9. 2025OmniSync: Towards Universal Lip Synchronization via Diffusion Transformerscited by 26
  10. 2025StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatarscited by 24
  11. 2025InterActHuman: Multi-Concept Human Animation with Layout-Aligned Audio Conditionscited by 20
  12. 2025Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesiscited by 20
  13. 2025DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformerscited by 19
  14. 2025AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinementcited by 19
  15. 2026DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generationcited by 19
  16. 2025MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generationcited by 18
  17. 2025FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformerscited by 17
  18. 2026Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversationcited by 17
  19. 2025MagicInfinite: Generating Infinite Talking Videos with Your Words and Voicecited by 16
  20. 2025Hallo4: High-Fidelity Dynamic Portrait Animation via Direct Preference Optimizationcited by 16
  21. 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 16
  22. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  23. 2025Audio-Sync Video Generation with Multi-Stream Temporal Controlcited by 14
  24. 2025AccVideo: Accelerating Video Diffusion Model with Synthetic Datasetcited by 13
  25. 2025SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformerscited by 13
  26. 2025HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animationcited by 13
  27. 2025X-UniMotion: Animating Human Images with Expressive, Unified and Identity-Agnostic Motion Latentscited by 13
  28. 2025Mitty: Diffusion-based Human-to-Robot Video Generationcited by 13
  29. 2025DreamVVT: Mastering Realistic Video Virtual Try-On in the Wild via a Stage-Wise Diffusion Transformer Frameworkcited by 12
  30. 2025ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modelingcited by 11
  31. 2025KlingAvatar 2.0 Technical Reportcited by 11
  32. 2025Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animationcited by 11
  33. 2026SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing modelcited by 11
  34. 2026LPM 1.0: Video-based Character Performance Modelcited by 11
  35. 2025Human-Centric Foundation Models: Perception, Generation and Agentic Modelingcited by 10
  36. 2025AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Modelscited by 10
  37. 2025RealisMotion: Decomposed Human Motion Control and Video Generation in the World Spacecited by 10
  38. 2025InfinityHuman: Towards Long-Term Audio-Driven Humancited by 9

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation
  2. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation
  3. 2025AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformers
  4. 2025TalkingPose: Efficient Face and Gesture Animation with Feedback-guided Diffusion Model
  5. 2024Real-time One-Step Diffusion-based Expressive Portrait Videos Generation
  6. 2025FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis
  7. 2026Mamba-Enhanced Implicit Motion Learning for Audio-Driven Portrait Animation
  8. 2025A Unit Enhancement and Guidance Framework for Audio-Driven Avatar Video Generation