# The DEVIL is in the Details: A Diagnostic Evaluation Benchmark for Video Inpainting

arXiv: https://arxiv.org/abs/2105.05332 · HTML: https://mlsanity.com/paper/2105.05332/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 11 May 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces](https://mlsanity.com/paper/1803.09179/) (2018)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- [Deep Video Inpainting](https://mlsanity.com/paper/1905.01639/) (2019)
- [Flow-edge Guided Video Completion](https://mlsanity.com/paper/2009.01835/) (2020)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [Video Inpainting by Jointly Learning Temporal Structure and Spatial Details](https://mlsanity.com/paper/1806.08482/) (2018)
- [Copy-and-Paste Networks for Deep Video Inpainting](https://mlsanity.com/paper/1908.11587/) (2019)
- [Onion-Peel Networks for Deep Video Completion](https://mlsanity.com/paper/1908.08718/) (2019)
- [Video Inpainting of Complex Scenes](https://mlsanity.com/paper/1503.05528/) (2015)
- [An Internal Learning Approach to Video Inpainting](https://mlsanity.com/paper/1909.07957/) (2019)
- [Characterizing and Improving Stability in Neural Style Transfer](https://mlsanity.com/paper/1705.02092/) (2017)
- [Learnable Gated Temporal Shift Module for Deep Video Inpainting](https://mlsanity.com/paper/1907.01131/) (2019)

## Built on by (most cited first, 12 of 12)

- [ImagerySearch: Adaptive Test-Time Search for Video Generation Beyond Semantic Dependency Constraints](https://mlsanity.com/paper/2510.14847/) (2025)
- [Semantically Consistent Video Inpainting with Conditional Diffusion Models](https://mlsanity.com/paper/2405.00251/) (2024)
- [Vision Transformer Based Video Hashing Retrieval for Tracing the Source of Fake Videos](https://mlsanity.com/paper/2112.08117/) (2021)
- [Expression-aware video inpainting for HMD removal in XR applications](https://mlsanity.com/paper/2401.14136/) (2024)
- [DynamicEval: Rethinking Evaluation for Dynamic Text-to-Video Synthesis](https://mlsanity.com/paper/2510.07441/) (2025)
- [Towards Unified Keyframe Propagation Models](https://mlsanity.com/paper/2205.09731/) (2022)
- [Reimagining Reality: A Comprehensive Survey of Video Inpainting Techniques](https://mlsanity.com/paper/2401.17883/) (2024)
- [Towards Realistic Landmark-Guided Facial Video Inpainting Based on GANs](https://mlsanity.com/paper/2402.09100/) (2024)
- [MVFNet: Multipurpose Video Forensics Network using Multiple Forms of Forensic Evidence](https://mlsanity.com/paper/2503.20991/) (2025)
- [Active Sampling for Ultra-Low-Bit-Rate Video Compression via Conditional Controlled Diffusion](https://mlsanity.com/paper/2605.02849/) (2026)
- [Copy What Is Seen, Generate What Is Not: Training-Free Anomaly-Aware Video Restoration](https://mlsanity.com/paper/2609.18836/) (2026)
- [MFVSR: A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal](https://mlsanity.com/paper/2610.18461/) (2026)

## Abstract (arXiv)

Quantitative evaluation has increased dramatically among recent video inpainting work, but the video and mask content used to gauge performance has received relatively little attention. Although attributes such as camera and background scene motion inherently change the difficulty of the task and affect methods differently, existing evaluation schemes fail to control for them, thereby providing minimal insight into inpainting failure modes. To address this gap, we propose the Diagnostic Evaluation of Video Inpainting on Landscapes (DEVIL) benchmark, which consists of two contributions: (i) a novel dataset of videos and masks labeled according to several key inpainting failure modes, and (ii) an evaluation scheme that samples slices of the dataset characterized by a fixed content attribute, and scores performance on each slice according to reconstruction, realism, and temporal consistency quality. By revealing systematic changes in performance induced by particular characteristics of the input content, our challenging benchmark enables more insightful analysis into video inpainting methods and serves as an invaluable diagnostic tool for the field. Our code and data are available at https://github.com/MichiganCOG/devil .