# A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insights

arXiv: https://arxiv.org/abs/2407.08428 · HTML: https://mlsanity.com/paper/2407.08428/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 98 of its 132 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 30 of 98)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)

## Built on by (most cited first, 25 of 25)

- [The Dawn of Video Generation: Preliminary Explorations with SORA-like Models](https://mlsanity.com/paper/2410.05227/) (2024)
- [Human Motion Video Generation: A Survey](https://mlsanity.com/paper/2509.03883/) (2025)
- [Generative Physical AI in Vision: A Survey](https://mlsanity.com/paper/2501.10928/) (2025)
- [Generative AI for Cel-Animation: A Survey](https://mlsanity.com/paper/2501.06250/) (2025)
- [Dormant: Defending against Pose-driven Human Image Animation](https://mlsanity.com/paper/2409.14424/) (2024)
- [Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook](https://mlsanity.com/paper/2411.19537/) (2024)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [AvatarShield: Visual Reinforcement Learning for Human-Centric Synthetic Video Detection](https://mlsanity.com/paper/2505.15173/) (2025)
- [Replace Anyone in Videos](https://mlsanity.com/paper/2409.19911/) (2024)
- [HumanSAM: Classifying Human-centric Forgery Videos in Human Spatial, Appearance, and Motion Anomaly](https://mlsanity.com/paper/2507.19924/) (2025)
- [A Survey: Spatiotemporal Consistency in Video Generation](https://mlsanity.com/paper/2502.17863/) (2025)
- [What Are You Doing? A Closer Look at Controllable Human Video Generation](https://mlsanity.com/paper/2503.04666/) (2025)
- [Applications and Advances of Artificial Intelligence in Music Generation:A Review](https://mlsanity.com/paper/2409.03715/) (2024)
- [Multi-human Interactive Talking Dataset](https://mlsanity.com/paper/2508.03050/) (2025)
- [Enhancing Fake News Video Detection via LLM-Driven Creative Process Simulation](https://mlsanity.com/paper/2510.04024/) (2025)
- [Human Video Generation from a Single Image with 3D Pose and View Control](https://mlsanity.com/paper/2602.21188/) (2026)
- [Training-free Motion Factorization for Compositional Video Generation](https://mlsanity.com/paper/2603.09104/) (2026)
- [Decoupled Video Generation with Chain of Training-free Diffusion Model Experts](https://mlsanity.com/paper/2408.13423/) (2024)
- [Collaborative Face Experts Fusion in Video Generation: Boosting Identity Consistency Across Large Face Poses](https://mlsanity.com/paper/2508.09476/) (2025)
- [Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascades](https://mlsanity.com/paper/2603.08028/) (2026)
- [FlashSign: Pose-Free Guidance for Efficient Sign Language Video Generation](https://mlsanity.com/paper/2603.27915/) (2026)
- [HumANDiff: Articulated Noise Diffusion for Motion-Consistent Human Video Generation](https://mlsanity.com/paper/2604.05961/) (2026)
- [SparseCtrl-HOI: Sparse Temporal Control for Human-Object Interaction Video Generation](https://mlsanity.com/paper/2607.05994/) (2026)
- [Video Generation Models: A Survey of Post-Training and Alignment](https://mlsanity.com/paper/2610.00812/) (2026)

## Abstract (arXiv)

Human video generation is a dynamic and rapidly evolving task that aims to synthesize 2D human body video sequences with generative models given control conditions such as text, audio, and pose. With the potential for wide-ranging applications in film, gaming, and virtual communication, the ability to generate natural and realistic human video is critical. Recent advancements in generative models have laid a solid foundation for the growing interest in this area. Despite the significant progress, the task of human video generation remains challenging due to the consistency of characters, the complexity of human motion, and difficulties in their relationship with the environment. This survey provides a comprehensive review of the current state of human video generation, marking, to the best of our knowledge, the first extensive literature review in this domain. We start with an introduction to the fundamentals of human video generation and the evolution of generative models that have facilitated the field's growth. We then examine the main methods employed for three key sub-tasks within human video generation: text-driven, audio-driven, and pose-driven motion generation. These areas are explored concerning the conditions that guide the generation process. Furthermore, we offer a collection of the most commonly utilized datasets and the evaluation metrics that are crucial in assessing the quality and realism of generated videos. The survey concludes with a discussion of the current challenges in the field and suggests possible directions for future research. The goal of this survey is to offer the research community a clear and holistic view of the advancements in human video generation, highlighting the milestones achieved and the challenges that lie ahead.