# PriorityCut: Occlusion-guided Regularization for Warp-based Image Animation

arXiv: https://arxiv.org/abs/2103.11600 · HTML: https://mlsanity.com/paper/2103.11600/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 32 references that are in the map.
- First posted on arXiv 22 Mar 2021.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018)
- [Self-Supervised Visual Planning with Temporal Skip Connections](https://mlsanity.com/paper/1710.05268/) (2017)
- [A U-Net Based Discriminator for Generative Adversarial Networks](https://mlsanity.com/paper/2002.12655/) (2020)
- [GridMask Data Augmentation](https://mlsanity.com/paper/2001.04086/) (2020)
- [MarioNETte: Few-shot Face Reenactment Preserving Identity of Unseen Targets](https://mlsanity.com/paper/1911.08139/) (2019)
- [Rethinking Data Augmentation for Image Super-resolution: A Comprehensive Analysis and a New Strategy](https://mlsanity.com/paper/2004.00448/) (2020)
- [F, B, Alpha Matting](https://mlsanity.com/paper/2003.07711/) (2020)
- [Hide-and-Seek: A Data Augmentation Technique for Weakly-Supervised Localization and Beyond](https://mlsanity.com/paper/1811.02545/) (2018)
- [Unsupervised Keypoint Learning for Guiding Class-Conditional Video Prediction](https://mlsanity.com/paper/1910.02027/) (2019)
- [Mesh Guided One-shot Face Reenactment using Graph Convolutional Networks](https://mlsanity.com/paper/2008.07783/) (2020)
- [Make a Face: Towards Arbitrary High Fidelity Face Manipulation](https://mlsanity.com/paper/1908.07191/) (2019)
- [Pyramidal Convolution: Rethinking Convolutional Neural Networks for Visual Recognition](https://mlsanity.com/paper/2006.11538/) (2020)
- [Dynamic Facial Expression Generation on Hilbert Hypersphere with Conditional Wasserstein Generative Adversarial Nets](https://mlsanity.com/paper/1907.10087/) (2019)
- [FReeNet: Multi-Identity Face Reenactment](https://mlsanity.com/paper/1905.11805/) (2019)
- [Cross-Identity Motion Transfer for Arbitrary Objects through Pose-Attentive Video Reassembling](https://mlsanity.com/paper/2007.08786/) (2020)
- [High Fidelity Face Manipulation with Extreme Poses and Expressions](https://mlsanity.com/paper/1903.12003/) (2019)

## Abstract (arXiv)

Image animation generates a video of a source image following the motion of a driving video. State-of-the-art self-supervised image animation approaches warp the source image according to the motion of the driving video and recover the warping artifacts by inpainting. These approaches mostly use vanilla convolution for inpainting, and vanilla convolution does not distinguish between valid and invalid pixels. As a result, visual artifacts are still noticeable after inpainting. CutMix is a state-of-the-art regularization strategy that cuts and mixes patches of images and is widely studied in different computer vision tasks. Among the remaining computer vision tasks, warp-based image animation is one of the fields that the effects of CutMix have yet to be studied. This paper first presents a preliminary study on the effects of CutMix on warp-based image animation. We observed in our study that CutMix helps improve only pixel values, but disturbs the spatial relationships between pixels. Based on such observation, we propose PriorityCut, a novel augmentation approach that uses the top-k percent occluded pixels of the foreground to regularize warp-based image animation. By leveraging the domain knowledge in warp-based image animation, PriorityCut significantly reduces the warping artifacts in state-of-the-art warp-based image animation models on diverse datasets.