# ProPainter: Improving Propagation and Transformer for Video Inpainting

arXiv: https://arxiv.org/abs/2309.03897 · HTML: https://mlsanity.com/paper/2309.03897/

## Key facts

- Cited by 197 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 46 references that are in the map.
- First posted on arXiv 7 Sep 2023.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://mlsanity.com/paper/2104.13371/) (2021)
- [BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond](https://mlsanity.com/paper/2012.02181/) (2020)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [Investigating Tradeoffs in Real-World Video Super-Resolution](https://mlsanity.com/paper/2111.12704/) (2021)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- [Towards An End-to-End Framework for Flow-Guided Video Inpainting](https://mlsanity.com/paper/2204.02663/) (2022)
- [Deep Video Inpainting](https://mlsanity.com/paper/1905.01639/) (2019)
- [Spatio-Temporal Filter Adaptive Network for Video Deblurring](https://mlsanity.com/paper/1904.12257/) (2019)
- [Flow-edge Guided Video Completion](https://mlsanity.com/paper/2009.01835/) (2020)
- [FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting](https://mlsanity.com/paper/2109.02974/) (2021)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [Flow-Guided Transformer for Video Inpainting](https://mlsanity.com/paper/2208.06768/) (2022)
- [Video Inpainting by Jointly Learning Temporal Structure and Spatial Details](https://mlsanity.com/paper/1806.08482/) (2018)
- [Copy-and-Paste Networks for Deep Video Inpainting](https://mlsanity.com/paper/1908.11587/) (2019)
- [Onion-Peel Networks for Deep Video Completion](https://mlsanity.com/paper/1908.08718/) (2019)
- [Short-Term and Long-Term Context Aggregation Network for Video Inpainting](https://mlsanity.com/paper/2009.05721/) (2020)
- [Progressive Temporal Feature Alignment Network for Video Inpainting](https://mlsanity.com/paper/2104.03507/) (2021)

## Built on by (most cited first, 30 of 197)

- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution](https://mlsanity.com/paper/2312.06640/) (2023)
- [VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Control](https://mlsanity.com/paper/2503.05639/) (2025)
- [MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling](https://mlsanity.com/paper/2409.16160/) (2024)
- [DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos](https://mlsanity.com/paper/2405.02280/) (2024)
- [Follow-Your-Creation: Empowering 4D Creation through Video Inpainting](https://mlsanity.com/paper/2506.04590/) (2025)
- [Dense Optical Tracking: Connecting the Dots](https://mlsanity.com/paper/2312.00786/) (2023)
- [Animate Anyone 2: High-Fidelity Character Image Animation with Environment Affordance](https://mlsanity.com/paper/2502.06145/) (2025)
- [Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists](https://mlsanity.com/paper/2502.06734/) (2025)
- [InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior](https://mlsanity.com/paper/2404.11613/) (2024)
- [IDOL: Instant Photorealistic 3D Human Creation from a Single Image](https://mlsanity.com/paper/2412.14963/) (2024)
- [Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models](https://mlsanity.com/paper/2606.17846/) (2026)
- [MiniMax-Remover: Taming Bad Noise Helps Video Object Removal](https://mlsanity.com/paper/2505.24873/) (2025)
- [Generative Omnimatte: Learning to Decompose Video into Layers](https://mlsanity.com/paper/2411.16683/) (2024)
- [ROSE: Remove Objects with Side Effects in Videos](https://mlsanity.com/paper/2508.18633/) (2025)
- [StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videos](https://mlsanity.com/paper/2409.07447/) (2024)
- [Generative Video Propagation](https://mlsanity.com/paper/2412.19761/) (2024)
- [VideoShield: Regulating Diffusion-based Video Generation Models via Watermarking](https://mlsanity.com/paper/2501.14195/) (2025)
- [Towards Language-Driven Video Inpainting via Multimodal Large Language Models](https://mlsanity.com/paper/2401.10226/) (2024)
- [MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing](https://mlsanity.com/paper/2408.08000/) (2024)
- [V2A-Mark: Versatile Deep Visual-Audio Watermarking for Manipulation Localization and Copyright Protection](https://mlsanity.com/paper/2404.16824/) (2024)
- [StableV2V: Stablizing Shape Consistency in Video-to-Video Editing](https://mlsanity.com/paper/2411.11045/) (2024)
- [VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editing](https://mlsanity.com/paper/2411.15260/) (2024)
- [MimicDreamer: Aligning Human and Robot Demonstrations for Scalable VLA Training](https://mlsanity.com/paper/2509.22199/) (2025)
- [MatAnyone: Stable Video Matting with Consistent Memory Propagation](https://mlsanity.com/paper/2501.14677/) (2025)
- [RORem: Training a Robust Object Remover with Human-in-the-Loop](https://mlsanity.com/paper/2501.00740/) (2025)
- [ReBot: Scaling Robot Learning with Real-to-Sim-to-Real Robotic Video Synthesis](https://mlsanity.com/paper/2503.14526/) (2025)
- [4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere](https://mlsanity.com/paper/2602.10094/) (2026)
- [Video Diffusion Models are Strong Video Inpainter](https://mlsanity.com/paper/2408.11402/) (2024)
- [VASE: Object-Centric Appearance and Shape Manipulation of Real Videos](https://mlsanity.com/paper/2401.02473/) (2024)

## Abstract (arXiv)

Flow-based propagation and spatiotemporal Transformer are two mainstream mechanisms in video inpainting (VI). Despite the effectiveness of these components, they still suffer from some limitations that affect their performance. Previous propagation-based approaches are performed separately either in the image or feature domain. Global image propagation isolated from learning may cause spatial misalignment due to inaccurate optical flow. Moreover, memory or computational constraints limit the temporal range of feature propagation and video Transformer, preventing exploration of correspondence information from distant frames. To address these issues, we propose an improved framework, called ProPainter, which involves enhanced ProPagation and an efficient Transformer. Specifically, we introduce dual-domain propagation that combines the advantages of image and feature warping, exploiting global correspondences reliably. We also propose a mask-guided sparse video Transformer, which achieves high efficiency by discarding unnecessary and redundant tokens. With these components, ProPainter outperforms prior arts by a large margin of 1.46 dB in PSNR while maintaining appealing efficiency.