# TCAN: Animating Human Images with Temporally Consistent Pose Guidance using Diffusion Models

arXiv: https://arxiv.org/abs/2407.09012 · HTML: https://mlsanity.com/paper/2407.09012/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 12 Jul 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 30)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16322/) (2023)
- [MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model](https://mlsanity.com/paper/2311.16498/) (2023)
- [Effective Whole-body Pose Estimation with Two-stages Distillation](https://mlsanity.com/paper/2307.15880/) (2023)
- [TryOnDiffusion: A Tale of Two UNets](https://mlsanity.com/paper/2306.08276/) (2023)
- [UPGPT: Universal Diffusion Model for Person Image Generation, Editing and Pose Transfer](https://mlsanity.com/paper/2304.08870/) (2023)
- [Transfer Learning for Pose Estimation of Illustrated Characters](https://mlsanity.com/paper/2108.01819/) (2021)
- [ViscoNet: Bridging and Harmonizing Visual and Textual Conditioning for ControlNet](https://mlsanity.com/paper/2312.03154/) (2023)

## Built on by (most cited first, 15 of 15)

- [Animate Anyone 2: High-Fidelity Character Image Animation with Environment Affordance](https://mlsanity.com/paper/2502.06145/) (2025)
- [Human Motion Video Generation: A Survey](https://mlsanity.com/paper/2509.03883/) (2025)
- [DreamDance: Animating Human Images by Enriching 3D Geometry Cues from 2D Poses](https://mlsanity.com/paper/2412.00397/) (2024)
- [HumanGif: Single-View Human Diffusion with Generative Prior](https://mlsanity.com/paper/2502.12080/) (2025)
- [SerialGen: Personalized Image Generation by First Standardization Then Personalization](https://mlsanity.com/paper/2412.01485/) (2024)
- [What Are You Doing? A Closer Look at Controllable Human Video Generation](https://mlsanity.com/paper/2503.04666/) (2025)
- [TV-LiVE: Training-Free, Text-Guided Video Editing via Layer Informed Vitality Exploitation](https://mlsanity.com/paper/2506.07205/) (2025)
- [SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation](https://mlsanity.com/paper/2509.24980/) (2025)
- [EverAnimate: Minute-Scale Human Animation via Latent Flow Restoration](https://mlsanity.com/paper/2605.15042/) (2026)
- [TalkingPose: Efficient Face and Gesture Animation with Feedback-guided Diffusion Model](https://mlsanity.com/paper/2512.00909/) (2025)
- [UniMo: Unifying 2D Video and 3D Human Motion with an Autoregressive Framework](https://mlsanity.com/paper/2512.03918/) (2025)
- [SwapAnyone: Consistent and Realistic Video Synthesis for Swapping Any Person into Any Video](https://mlsanity.com/paper/2503.09154/) (2025)
- [Generating Fit Check Videos with a Handheld Camera](https://mlsanity.com/paper/2505.23886/) (2025)
- [A Study on Real-time Object Detection using Deep Learning](https://mlsanity.com/paper/2602.15926/) (2026)
- [MultiAnimate: A Unified Framework for Controllable Multi-Character Animation](https://mlsanity.com/paper/2607.13415/) (2026)

## Abstract (arXiv)

Pose-driven human-image animation diffusion models have shown remarkable capabilities in realistic human video synthesis. Despite the promising results achieved by previous approaches, challenges persist in achieving temporally consistent animation and ensuring robustness with off-the-shelf pose detectors. In this paper, we present TCAN, a pose-driven human image animation method that is robust to erroneous poses and consistent over time. In contrast to previous methods, we utilize the pre-trained ControlNet without fine-tuning to leverage its extensive pre-acquired knowledge from numerous pose-image-caption pairs. To keep the ControlNet frozen, we adapt LoRA to the UNet layers, enabling the network to align the latent space between the pose and appearance features. Additionally, by introducing an additional temporal layer to the ControlNet, we enhance robustness against outliers of the pose detector. Through the analysis of attention maps over the temporal axis, we also designed a novel temperature map leveraging pose information, allowing for a more static background. Extensive experiments demonstrate that the proposed method can achieve promising results in video synthesis tasks encompassing various poses, like chibi. Project Page: https://eccv2024tcan.github.io/