# Human4DiT: 360-degree Human Video Generation with 4D Diffusion Transformer

arXiv: https://arxiv.org/abs/2405.17405 · HTML: https://mlsanity.com/paper/2405.17405/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 24 of its 51 references that are in the map.
- First posted on arXiv 27 May 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 24 of 24)

- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [All are Worth Words: A ViT Backbone for Diffusion Models](https://mlsanity.com/paper/2209.12152/) (2022)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors](https://mlsanity.com/paper/2105.01859/) (2021)
- [BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion](https://mlsanity.com/paper/2306.16940/) (2023)
- [Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset](https://mlsanity.com/paper/2307.00818/) (2023)
- [Generating Holistic 3D Human Motion from Speech](https://mlsanity.com/paper/2212.04420/) (2022)
- [Fast Training of Diffusion Models with Masked Transformers](https://mlsanity.com/paper/2306.09305/) (2023)
- [Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion](https://mlsanity.com/paper/2402.03162/) (2024)
- [Deformable GANs for Pose-based Human Image Generation](https://mlsanity.com/paper/1801.00055/) (2017)
- [VDT: General-purpose Video Diffusion Transformers via Mask Modeling](https://mlsanity.com/paper/2305.13311/) (2023)
- [DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity Human-centric Rendering](https://mlsanity.com/paper/2307.10173/) (2023)
- [Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models](https://mlsanity.com/paper/2312.01409/) (2023)
- [Appearance and Pose-Conditioned Human Image Generation using Deformable GANs](https://mlsanity.com/paper/1905.00007/) (2019)
- [CloSET: Modeling Clothed Humans on Continuous Surface with Explicit Template Decomposition](https://mlsanity.com/paper/2304.03167/) (2023)

## Built on by (most cited first, 30 of 61)

- [LHM: Large Animatable Human Reconstruction Model from a Single Image in Seconds](https://mlsanity.com/paper/2503.10625/) (2025)
- [Diffusion Self-Distillation for Zero-Shot Customized Image Generation](https://mlsanity.com/paper/2411.18616/) (2024)
- [HumanDiT: Pose-Guided Diffusion Transformer for Long-form Human Motion Video Generation](https://mlsanity.com/paper/2502.04847/) (2025)
- [AniGS: Animatable Gaussian Avatar from a Single Image with Inconsistent Gaussian Reconstruction](https://mlsanity.com/paper/2412.02684/) (2024)
- [DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidance](https://mlsanity.com/paper/2504.01724/) (2025)
- [Reangle-A-Video: 4D Video Generation as Video-to-Video Translation](https://mlsanity.com/paper/2503.09151/) (2025)
- [Lodge++: High-quality and Long Dance Generation with Vivid Choreography Patterns](https://mlsanity.com/paper/2410.20389/) (2024)
- [ShapeGen4D: Towards High Quality 4D Shape Generation from Videos](https://mlsanity.com/paper/2510.06208/) (2025)
- [ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Grasping](https://mlsanity.com/paper/2412.16212/) (2024)
- [DreamDance: Animating Human Images by Enriching 3D Geometry Cues from 2D Poses](https://mlsanity.com/paper/2412.00397/) (2024)
- [MVP4D: Multi-View Portrait Video Diffusion for Animatable 4D Avatars](https://mlsanity.com/paper/2510.12785/) (2025)
- [MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation](https://mlsanity.com/paper/2503.19383/) (2025)
- [HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation](https://mlsanity.com/paper/2506.08797/) (2025)
- [Human-Centric Foundation Models: Perception, Generation and Agentic Modeling](https://mlsanity.com/paper/2502.08556/) (2025)
- [RealisMotion: Decomposed Human Motion Control and Video Generation in the World Space](https://mlsanity.com/paper/2508.08588/) (2025)
- [Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priors](https://mlsanity.com/paper/2411.17249/) (2024)
- [UP2You: Fast Reconstruction of Yourself from Unconstrained Photo Collections](https://mlsanity.com/paper/2509.24817/) (2025)
- [HumanRAM: Feed-forward Human Reconstruction and Animation Model using Transformers](https://mlsanity.com/paper/2506.03118/) (2025)
- [CharacterShot: Controllable and Consistent 4D Character Animation](https://mlsanity.com/paper/2508.07409/) (2025)
- [HumanGif: Single-View Human Diffusion with Generative Prior](https://mlsanity.com/paper/2502.12080/) (2025)
- [Vid-CamEdit: Video Camera Trajectory Editing with Generative Rendering from Estimated Geometry](https://mlsanity.com/paper/2506.13697/) (2025)
- [AnimaMimic: Imitating 3D Animation from Video Priors](https://mlsanity.com/paper/2512.14133/) (2025)
- [LHM++: An Efficient Large Human Reconstruction Model for Pose-free Images to 3D](https://mlsanity.com/paper/2506.13766/) (2025)
- [Free-viewpoint Human Animation with Pose-correlated Reference Selection](https://mlsanity.com/paper/2412.17290/) (2024)
- [iDiT-HOI: Inpainting-based Hand Object Interaction Reenactment via Video Diffusion Transformer](https://mlsanity.com/paper/2506.12847/) (2025)
- [MV-Performer: Taming Video Diffusion Model for Faithful and Synchronized Multi-view Performer Synthesis](https://mlsanity.com/paper/2510.07190/) (2025)
- [HarmoHOI: Harmonizing Appearance and 3D Motion for Multi-view Hand-Object Interaction Synthesis](https://mlsanity.com/paper/2607.17097/) (2026)
- [CFSynthesis: Controllable and Free-view 3D Human Video Synthesis](https://mlsanity.com/paper/2412.11067/) (2024)
- [EverybodyDance: Bipartite Graph-Based Identity Correspondence for Multi-Character Animation](https://mlsanity.com/paper/2512.16360/) (2025)
- [MotionWeaver: Holistic 4D-Anchored Framework for Multi-Humanoid Image Animation](https://mlsanity.com/paper/2602.13326/) (2026)

## Abstract (arXiv)

We present a novel approach for generating 360-degree high-quality, spatio-temporally coherent human videos from a single image. Our framework combines the strengths of diffusion transformers for capturing global correlations across viewpoints and time, and CNNs for accurate condition injection. The core is a hierarchical 4D transformer architecture that factorizes self-attention across views, time steps, and spatial dimensions, enabling efficient modeling of the 4D space. Precise conditioning is achieved by injecting human identity, camera parameters, and temporal signals into the respective transformers. To train this model, we collect a multi-dimensional dataset spanning images, videos, multi-view data, and limited 4D footage, along with a tailored multi-dimensional training strategy. Our approach overcomes the limitations of previous methods based on generative adversarial networks or vanilla diffusion models, which struggle with complex motions, viewpoint changes, and generalization. Through extensive experiments, we demonstrate our method's ability to synthesize 360-degree realistic, coherent human motion videos, paving the way for advanced multimedia applications in areas such as virtual reality and animation.