# Progressive Temporal Feature Alignment Network for Video Inpainting

arXiv: https://arxiv.org/abs/2104.03507 · HTML: https://mlsanity.com/paper/2104.03507/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 8 Apr 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- [YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video](https://mlsanity.com/paper/1702.00824/) (2017)
- [Flow-edge Guided Video Completion](https://mlsanity.com/paper/2009.01835/) (2020)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [The 2018 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1803.00557/) (2018)
- [Onion-Peel Networks for Deep Video Completion](https://mlsanity.com/paper/1908.08718/) (2019)
- [Learning Spatiotemporal Features with 3D Convolutional Networks](https://mlsanity.com/paper/1412.0767/) (2014)
- [Video Inpainting of Complex Scenes](https://mlsanity.com/paper/1503.05528/) (2015)
- [Short-Term and Long-Term Context Aggregation Network for Video Inpainting](https://mlsanity.com/paper/2009.05721/) (2020)
- [Self-supervised Learning for Video Correspondence Flow](https://mlsanity.com/paper/1905.00875/) (2019)
- [Learnable Gated Temporal Shift Module for Deep Video Inpainting](https://mlsanity.com/paper/1907.01131/) (2019)

## Built on by (most cited first, 30 of 35)

- [ProPainter: Improving Propagation and Transformer for Video Inpainting](https://mlsanity.com/paper/2309.03897/) (2023)
- [Towards An End-to-End Framework for Flow-Guided Video Inpainting](https://mlsanity.com/paper/2204.02663/) (2022)
- [VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Control](https://mlsanity.com/paper/2503.05639/) (2025)
- [Flow-Guided Transformer for Video Inpainting](https://mlsanity.com/paper/2208.06768/) (2022)
- [ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation](https://mlsanity.com/paper/2308.00906/) (2023)
- [Video Event Restoration Based on Keyframes for Video Anomaly Detection](https://mlsanity.com/paper/2304.05112/) (2023)
- [Deep Learning-based Image and Video Inpainting: A Survey](https://mlsanity.com/paper/2401.03395/) (2024)
- [ROSE: Remove Objects with Side Effects in Videos](https://mlsanity.com/paper/2508.18633/) (2025)
- [Video Diffusion Models are Strong Video Inpainter](https://mlsanity.com/paper/2408.11402/) (2024)
- [Error Compensation Framework for Flow-Guided Video Inpainting](https://mlsanity.com/paper/2207.10391/) (2022)
- [DeViT: Deformed Vision Transformers in Video Inpainting](https://mlsanity.com/paper/2209.13925/) (2022)
- [Fast-Vid2Vid: Spatial-Temporal Compression for Video-to-Video Synthesis](https://mlsanity.com/paper/2207.05049/) (2022)
- [PISCO: Precise Video Instance Insertion with Sparse Control](https://mlsanity.com/paper/2602.08277/) (2026)
- [Semi-Supervised Video Inpainting with Cycle Consistency Constraints](https://mlsanity.com/paper/2208.06807/) (2022)
- [Deep Face Video Inpainting via UV Mapping](https://mlsanity.com/paper/2109.00681/) (2021)
- [Exploiting Optical Flow Guidance for Transformer-Based Video Inpainting](https://mlsanity.com/paper/2301.10048/) (2023)
- [Deficiency-Aware Masked Transformer for Video Inpainting](https://mlsanity.com/paper/2307.08629/) (2023)
- [From Ideal to Real: Stable Video Object Removal under Imperfect Conditions](https://mlsanity.com/paper/2603.09283/) (2026)
- [Exemplar-based Video Colorization with Long-term Spatiotemporal Dependency](https://mlsanity.com/paper/2303.15081/) (2023)
- [3D StreetUnveiler with Semantic-aware 2DGS -- a simple baseline](https://mlsanity.com/paper/2405.18416/) (2024)
- [BVINet: Unlocking Blind Video Inpainting with Zero Annotations](https://mlsanity.com/paper/2502.01181/) (2025)
- [One-Shot Video Inpainting](https://mlsanity.com/paper/2302.14362/) (2023)
- [Towards Online Real-Time Memory-based Video Inpainting Transformers](https://mlsanity.com/paper/2403.16161/) (2024)
- [IPAD: Industrial Process Anomaly Detection Dataset](https://mlsanity.com/paper/2404.15033/) (2024)
- [Spatial-Temporal Residual Aggregation for High Resolution Video Inpainting](https://mlsanity.com/paper/2111.03574/) (2021)
- [Expression-aware video inpainting for HMD removal in XR applications](https://mlsanity.com/paper/2401.14136/) (2024)
- [Raformer: Redundancy-Aware Transformer for Video Wire Inpainting](https://mlsanity.com/paper/2404.15802/) (2024)
- [Learning Stochastic Bridges for Video Object Removal via Video-to-Video Translation](https://mlsanity.com/paper/2601.12066/) (2026)
- [E2V-SDE: From Asynchronous Events to Fast and Continuous Video Reconstruction via Neural Stochastic Differential Equations](https://mlsanity.com/paper/2206.07578/) (2022)
- [Towards Realistic Landmark-Guided Facial Video Inpainting Based on GANs](https://mlsanity.com/paper/2402.09100/) (2024)

## Abstract (arXiv)

Video inpainting aims to fill spatio-temporal "corrupted" regions with plausible content. To achieve this goal, it is necessary to find correspondences from neighbouring frames to faithfully hallucinate the unknown content. Current methods achieve this goal through attention, flow-based warping, or 3D temporal convolution. However, flow-based warping can create artifacts when optical flow is not accurate, while temporal convolution may suffer from spatial misalignment. We propose 'Progressive Temporal Feature Alignment Network', which progressively enriches features extracted from the current frame with the feature warped from neighbouring frames using optical flow. Our approach corrects the spatial misalignment in the temporal feature propagation stage, greatly improving visual quality and temporal consistency of the inpainted videos. Using the proposed architecture, we achieve state-of-the-art performance on the DAVIS and FVI datasets compared to existing deep learning approaches. Code is available at https://github.com/MaureenZOU/TSAM.