# UniAnimate-DiT: Human Image Animation with Large-Scale Video Diffusion Transformer

arXiv: https://arxiv.org/abs/2504.11289 · HTML: https://mlsanity.com/paper/2504.11289/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 14 of its 16 references that are in the map.
- First posted on arXiv 15 Apr 2025.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 14 of 14)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models](https://mlsanity.com/paper/2311.04145/) (2023)
- [MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model](https://mlsanity.com/paper/2311.16498/) (2023)
- [Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance](https://mlsanity.com/paper/2403.14781/) (2024)
- [UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation](https://mlsanity.com/paper/2406.01188/) (2024)
- [Animate-X: Universal Character Image Animation with Enhanced Motion Representation](https://mlsanity.com/paper/2410.10306/) (2024)
- [Pose Guided Human Video Generation](https://mlsanity.com/paper/1807.11152/) (2018)
- [A Recipe for Scaling up Text-to-Video Generation with Text-free Videos](https://mlsanity.com/paper/2312.15770/) (2023)
- [Exploring the Evolution of Physics Cognition in Video Generation: A Survey](https://mlsanity.com/paper/2503.21765/) (2025)

## Built on by (most cited first, 30 of 42)

- [VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models](https://mlsanity.com/paper/2505.23656/) (2025)
- [Wan-Animate: Unified Character Animation and Replacement with Holistic Replication](https://mlsanity.com/paper/2509.14055/) (2025)
- [Stable Video Infinity: Infinite-Length Video Generation with Error Recycling](https://mlsanity.com/paper/2510.09212/) (2025)
- [HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation](https://mlsanity.com/paper/2506.08797/) (2025)
- [DanceTogether! Identity-Preserving Multi-Person Interactive Video Generation](https://mlsanity.com/paper/2505.18078/) (2025)
- [X-UniMotion: Animating Human Images with Expressive, Unified and Identity-Agnostic Motion Latents](https://mlsanity.com/paper/2508.09383/) (2025)
- [SCAIL: Towards Studio-Grade Character Animation via In-Context Learning of 3D-Consistent Pose Representations](https://mlsanity.com/paper/2512.05905/) (2025)
- [AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models](https://mlsanity.com/paper/2505.20255/) (2025)
- [Animate-X++: Universal Character Image Animation with Dynamic Backgrounds](https://mlsanity.com/paper/2508.09454/) (2025)
- [SteadyDancer: Harmonized and Coherent Human Image Animation with First-Frame Preservation](https://mlsanity.com/paper/2511.19320/) (2025)
- [PoseGen: In-Context LoRA Finetuning for Pose-Controllable Long Human Video Generation](https://mlsanity.com/paper/2508.05091/) (2025)
- [One-to-All Animation: Alignment-Free Character Animation and Image Pose Transfer](https://mlsanity.com/paper/2511.22940/) (2025)
- [SCAIL-2: Unifying Controlled Character Animation with End-to-End In-Context Conditioning](https://mlsanity.com/paper/2606.10804/) (2026)
- [OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer](https://mlsanity.com/paper/2601.14250/) (2026)
- [iDiT-HOI: Inpainting-based Hand Object Interaction Reenactment via Video Diffusion Transformer](https://mlsanity.com/paper/2506.12847/) (2025)
- [DreamActor-M2: Universal Character Image Animation via Spatiotemporal In-Context Learning](https://mlsanity.com/paper/2601.21716/) (2026)
- [Generating Human Motion Videos using a Cascaded Text-to-Video Framework](https://mlsanity.com/paper/2510.03909/) (2025)
- [Kling-MotionControl Technical Report](https://mlsanity.com/paper/2603.03160/) (2026)
- [MTVCraft: Tokenizing 4D Motion for Arbitrary Character Animation](https://mlsanity.com/paper/2505.10238/) (2025)
- [CoDance: An Unbind-Rebind Paradigm for Robust Multi-Subject Animation](https://mlsanity.com/paper/2601.11096/) (2026)
- [Making Avatars Interact: Towards Text-Driven Human-Object Interaction for Controllable Talking Avatars](https://mlsanity.com/paper/2602.01538/) (2026)
- [EverAnimate: Minute-Scale Human Animation via Latent Flow Restoration](https://mlsanity.com/paper/2605.15042/) (2026)
- [Image-to-Video Diffusion: From Foundations to Open Frontiers](https://mlsanity.com/paper/2605.17248/) (2026)
- [ControlHair: Synergizing Physics Simulator and Video Diffusion for Controllable Dynamic Hair Rendering](https://mlsanity.com/paper/2509.21541/) (2025)
- [AHOY! Animatable Humans under Occlusion from YouTube Videos with Gaussian Splatting and Video Diffusion Priors](https://mlsanity.com/paper/2603.17975/) (2026)
- [StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation](https://mlsanity.com/paper/2607.20174/) (2026)
- [In-Video Instructions: Visual Signals as Generative Control](https://mlsanity.com/paper/2511.19401/) (2025)
- [High-Fidelity and Long-Duration Human Image Animation with Diffusion Transformer](https://mlsanity.com/paper/2512.21905/) (2025)
- [IM-Animation: An Implicit Motion Representation for Identity-decoupled Character Animation](https://mlsanity.com/paper/2602.07498/) (2026)
- [GenHOI: Towards Object-Consistent Hand-Object Interaction with Temporally Balanced and Spatially Selective Object Injection](https://mlsanity.com/paper/2603.06048/) (2026)

## Abstract (arXiv)

This report presents UniAnimate-DiT, an advanced project that leverages the cutting-edge and powerful capabilities of the open-source Wan2.1 model for consistent human image animation. Specifically, to preserve the robust generative capabilities of the original Wan2.1 model, we implement Low-Rank Adaptation (LoRA) technique to fine-tune a minimal set of parameters, significantly reducing training memory overhead. A lightweight pose encoder consisting of multiple stacked 3D convolutional layers is designed to encode motion information of driving poses. Furthermore, we adopt a simple concatenation operation to integrate the reference appearance into the model and incorporate the pose information of the reference image for enhanced pose alignment. Experimental results show that our approach achieves visually appearing and temporally consistent high-fidelity animations. Trained on 480p (832x480) videos, UniAnimate-DiT demonstrates strong generalization capabilities to seamlessly upscale to 720P (1280x720) during inference. The training and inference code is publicly available at https://github.com/ali-vilab/UniAnimate-DiT.