# Semi-Supervised Video Inpainting with Cycle Consistency Constraints

arXiv: https://arxiv.org/abs/2208.06807 · HTML: https://mlsanity.com/paper/2208.06807/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 53 references that are in the map.
- First posted on arXiv 14 Aug 2022.

## Builds on (most cited first, 30 of 38)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [Video Object Segmentation with Episodic Graph Memory Networks](https://mlsanity.com/paper/2007.07020/) (2020)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- [Towards An End-to-End Framework for Flow-Guided Video Inpainting](https://mlsanity.com/paper/2204.02663/) (2022)
- [Flow-edge Guided Video Completion](https://mlsanity.com/paper/2009.01835/) (2020)
- [FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting](https://mlsanity.com/paper/2109.02974/) (2021)
- [Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting](https://mlsanity.com/paper/1904.07475/) (2019)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [Efficient Regional Memory Network for Video Object Segmentation](https://mlsanity.com/paper/2103.12934/) (2021)
- [Image Inpainting via Generative Multi-column Convolutional Neural Networks](https://mlsanity.com/paper/1810.08771/) (2018)
- [Learning Fast and Robust Target Models for Video Object Segmentation](https://mlsanity.com/paper/2003.00908/) (2020)
- [Hierarchical Memory Matching Network for Video Object Segmentation](https://mlsanity.com/paper/2109.11404/) (2021)
- [RANet: Ranking Attention Network for Fast Video Object Segmentation](https://mlsanity.com/paper/1908.06647/) (2019)
- [Video Inpainting by Jointly Learning Temporal Structure and Spatial Details](https://mlsanity.com/paper/1806.08482/) (2018)
- [Copy-and-Paste Networks for Deep Video Inpainting](https://mlsanity.com/paper/1908.11587/) (2019)
- [State-Aware Tracker for Real-Time Video Object Segmentation](https://mlsanity.com/paper/2003.00482/) (2020)
- [Onion-Peel Networks for Deep Video Completion](https://mlsanity.com/paper/1908.08718/) (2019)
- [WaveFill: A Wavelet-based Generation Network for Image Inpainting](https://mlsanity.com/paper/2107.11027/) (2021)
- [Video Inpainting of Complex Scenes](https://mlsanity.com/paper/1503.05528/) (2015)
- [Fast Video Object Segmentation With Temporal Aggregation Network and Dynamic Template Matching](https://mlsanity.com/paper/2007.05687/) (2020)
- [Short-Term and Long-Term Context Aggregation Network for Video Inpainting](https://mlsanity.com/paper/2009.05721/) (2020)
- [Progressive Temporal Feature Alignment Network for Video Inpainting](https://mlsanity.com/paper/2104.03507/) (2021)
- [An Internal Learning Approach to Video Inpainting](https://mlsanity.com/paper/1909.07957/) (2019)
- [DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation](https://mlsanity.com/paper/1909.12471/) (2019)

## Built on by (most cited first, 8 of 8)

- [Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos](https://mlsanity.com/paper/2305.16301/) (2023)
- [Micro-gesture Online Recognition using Learnable Query Points](https://mlsanity.com/paper/2407.04490/) (2024)
- [QualiTeacher: Quality-Conditioned Pseudo-Labeling for Real-World Image Restoration](https://mlsanity.com/paper/2603.08030/) (2026)
- [BVINet: Unlocking Blind Video Inpainting with Zero Annotations](https://mlsanity.com/paper/2502.01181/) (2025)
- [Raformer: Redundancy-Aware Transformer for Video Wire Inpainting](https://mlsanity.com/paper/2404.15802/) (2024)
- [Prompt-Aware Controllable Shadow Removal](https://mlsanity.com/paper/2501.15043/) (2025)
- [VIRTUS-FPP: Virtual Sensor Modeling for Fringe Projection Profilometry in NVIDIA Isaac Sim](https://mlsanity.com/paper/2509.22685/) (2025)
- [Blind Bitstream-corrupted Video Recovery via Metadata-guided Diffusion Model](https://mlsanity.com/paper/2604.13906/) (2026)

## Abstract (arXiv)

Deep learning-based video inpainting has yielded promising results and gained increasing attention from researchers. Generally, these methods usually assume that the corrupted region masks of each frame are known and easily obtained. However, the annotation of these masks are labor-intensive and expensive, which limits the practical application of current methods. Therefore, we expect to relax this assumption by defining a new semi-supervised inpainting setting, making the networks have the ability of completing the corrupted regions of the whole video using the annotated mask of only one frame. Specifically, in this work, we propose an end-to-end trainable framework consisting of completion network and mask prediction network, which are designed to generate corrupted contents of the current frame using the known mask and decide the regions to be filled of the next frame, respectively. Besides, we introduce a cycle consistency loss to regularize the training parameters of these two networks. In this way, the completion network and the mask prediction network can constrain each other, and hence the overall performance of the trained model can be maximized. Furthermore, due to the natural existence of prior knowledge (e.g., corrupted contents and clear borders), current video inpainting datasets are not suitable in the context of semi-supervised video inpainting. Thus, we create a new dataset by simulating the corrupted video of real-world scenarios. Extensive experimental results are reported to demonstrate the superiority of our model in the video inpainting task. Remarkably, although our model is trained in a semi-supervised manner, it can achieve comparable performance as fully-supervised methods.