# Motion Transformer for Unsupervised Image Animation

arXiv: https://arxiv.org/abs/2209.14024 · HTML: https://mlsanity.com/paper/2209.14024/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 63 references that are in the map.
- First posted on arXiv 28 Sep 2022.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [FSGAN: Subject Agnostic Face Swapping and Reenactment](https://mlsanity.com/paper/1908.05932/) (2019)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers](https://mlsanity.com/paper/2108.08839/) (2021)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [Pose Guided Person Image Generation](https://mlsanity.com/paper/1705.09368/) (2017)
- [Multi-Garment Net: Learning to Dress 3D People from Images](https://mlsanity.com/paper/1908.06903/) (2019)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018)
- [Deformable GANs for Pose-based Human Image Generation](https://mlsanity.com/paper/1801.00055/) (2017)
- [GANimation: Anatomically-aware Facial Animation from a Single Image](https://mlsanity.com/paper/1807.09251/) (2018)
- [Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis](https://mlsanity.com/paper/1909.12224/) (2019)
- [Progressive Pose Attention Transfer for Person Image Generation](https://mlsanity.com/paper/1904.03349/) (2019)
- [High-Fidelity Pluralistic Image Completion with Transformers](https://mlsanity.com/paper/2103.14031/) (2021)
- [TransPose: Keypoint Localization via Transformer](https://mlsanity.com/paper/2012.14214/) (2020)
- [Diverse Part Discovery: Occluded Person Re-identification with Part-Aware Transformer](https://mlsanity.com/paper/2106.04095/) (2021)

## Built on by (most cited first, 8 of 8)

- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding](https://mlsanity.com/paper/2405.03121/) (2024)
- [Learning Motion Refinement for Unsupervised Face Animation](https://mlsanity.com/paper/2310.13912/) (2023)
- [Generative Human Video Compression with Multi-granularity Temporal Trajectory Factorization](https://mlsanity.com/paper/2410.10171/) (2024)
- [SPOTR: Spatio-temporal Pose Transformers for Human Motion Prediction](https://mlsanity.com/paper/2303.06277/) (2023)
- [AnaMoDiff: 2D Analogical Motion Diffusion via Disentangled Denoising](https://mlsanity.com/paper/2402.03549/) (2024)
- [Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based Spatiotemporal Diffusion for Audio-driven Talking Portrait](https://mlsanity.com/paper/2503.12963/) (2025)
- [One-shot Human Motion Transfer via Occlusion-Robust Flow Prediction and Neural Texturing](https://mlsanity.com/paper/2412.06174/) (2024)

## Abstract (arXiv)

Image animation aims to animate a source image by using motion learned from a driving video. Current state-of-the-art methods typically use convolutional neural networks (CNNs) to predict motion information, such as motion keypoints and corresponding local transformations. However, these CNN based methods do not explicitly model the interactions between motions; as a result, the important underlying motion relationship may be neglected, which can potentially lead to noticeable artifacts being produced in the generated animation video. To this end, we propose a new method, the motion transformer, which is the first attempt to build a motion estimator based on a vision transformer. More specifically, we introduce two types of tokens in our proposed method: i) image tokens formed from patch features and corresponding position encoding; and ii) motion tokens encoded with motion information. Both types of tokens are sent into vision transformers to promote underlying interactions between them through multi-head self attention blocks. By adopting this process, the motion information can be better learned to boost the model performance. The final embedded motion tokens are then used to predict the corresponding motion keypoints and local transformations. Extensive experiments on benchmark datasets show that our proposed method achieves promising results to the state-of-the-art baselines. Our source code will be public available.