# Differential Motion Evolution for Fine-Grained Motion Deformation in Unsupervised Image Animation

arXiv: https://arxiv.org/abs/2110.04658 · HTML: https://mlsanity.com/paper/2110.04658/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 72 references that are in the map.
- First posted on arXiv 9 Oct 2021.

## Builds on (most cited first, 30 of 47)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [Multi-Garment Net: Learning to Dress 3D People from Images](https://mlsanity.com/paper/1908.06903/) (2019)
- [Joint Discriminative and Generative Learning for Person Re-identification](https://mlsanity.com/paper/1904.07223/) (2019)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Few-shot Video-to-Video Synthesis](https://mlsanity.com/paper/1910.12713/) (2019)
- [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018)
- [Deep Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/1804.04272/) (2018)
- [View Synthesis by Appearance Flow](https://mlsanity.com/paper/1605.03557/) (2016)
- [Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis](https://mlsanity.com/paper/1909.12224/) (2019)
- [TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style](https://mlsanity.com/paper/2003.04583/) (2020)
- [Self-Supervised Visual Planning with Temporal Skip Connections](https://mlsanity.com/paper/1710.05268/) (2017)
- [The DeepFake Detection Challenge (DFDC) Dataset](https://mlsanity.com/paper/2006.07397/) (2020)
- [Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatars](https://mlsanity.com/paper/2008.10174/) (2020)
- [DwNet: Dense warp-based network for pose-guided human video generation](https://mlsanity.com/paper/1910.09139/) (2019)
- [Unsupervised Learning of Object Landmarks through Conditional Image Generation](https://mlsanity.com/paper/1806.07823/) (2018)

## Built on by (most cited first, 3 of 3)

- [Brain-ID: Learning Contrast-agnostic Anatomical Representations for Brain Imaging](https://mlsanity.com/paper/2311.16914/) (2023)
- [PEPSI: Pathology-Enhanced Pulse-Sequence-Invariant Representations for Brain MRI](https://mlsanity.com/paper/2403.06227/) (2024)
- [Efficient conditioned face animation using frontally-viewed embedding](https://mlsanity.com/paper/2203.08765/) (2022)

## Abstract (arXiv)

Image animation is the task of transferring the motion of a driving video to a given object in a source image. While great progress has recently been made in unsupervised motion transfer, requiring no labeled data or domain priors, many current unsupervised approaches still struggle to capture the motion deformations when large motion/view discrepancies occur between the source and driving domains. Under such conditions, there is simply not enough information to capture the motion field properly. We introduce DiME (Differential Motion Evolution), an end-to-end unsupervised motion transfer framework integrating differential refinement for motion estimation. Key findings are twofold: (1) by capturing the motion transfer with an ordinary differential equation (ODE), it helps to regularize the motion field, and (2) by utilizing the source image itself, we are able to inpaint occluded/missing regions arising from large motion changes. Additionally, we also propose a natural extension to the ODE idea, which is that DiME can easily leverage multiple different views of the source object whenever they are available by modeling an ODE per view. Extensive experiments across 9 benchmarks show DiME outperforms the state-of-the-arts by a significant margin and generalizes much better to unseen objects.