# Image Inpainting by End-to-End Cascaded Refinement with Mask Awareness

arXiv: https://arxiv.org/abs/2104.13743 · HTML: https://mlsanity.com/paper/2104.13743/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 59 references that are in the map.
- First posted on arXiv 28 Apr 2021.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform](https://mlsanity.com/paper/1804.02815/) (2018)
- [Multi-Scale Progressive Fusion Network for Single Image Deraining](https://mlsanity.com/paper/2003.10985/) (2020)
- [Deep Back-Projection Networks For Super-Resolution](https://mlsanity.com/paper/1803.02735/) (2018)
- [EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning](https://mlsanity.com/paper/1901.00212/) (2019)
- [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019)
- [Semantic Image Inpainting with Deep Generative Models](https://mlsanity.com/paper/1607.07539/) (2016)
- [A Model-driven Deep Neural Network for Single Image Rain Removal](https://mlsanity.com/paper/2005.01333/) (2020)
- [StructureFlow: Image Inpainting via Structure-aware Appearance Flow](https://mlsanity.com/paper/1908.03852/) (2019)
- [Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting](https://mlsanity.com/paper/1904.07475/) (2019)
- [High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis](https://mlsanity.com/paper/1611.09969/) (2016)
- [Foreground-aware Image Inpainting](https://mlsanity.com/paper/1901.05945/) (2019)
- [Shift-Net: Image Inpainting via Deep Feature Rearrangement](https://mlsanity.com/paper/1801.09392/) (2018)
- [Image Inpainting via Generative Multi-column Convolutional Neural Networks](https://mlsanity.com/paper/1810.08771/) (2018)
- [Image Inpainting with Learnable Bidirectional Attention Maps](https://mlsanity.com/paper/1909.00968/) (2019)
- [SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting](https://mlsanity.com/paper/1805.03356/) (2018)
- [Contextual-based Image Inpainting: Infer, Match, and Translate](https://mlsanity.com/paper/1711.08590/) (2017)
- [DRD-Net: Detail-recovery Image Deraining via Context Aggregation Networks](https://mlsanity.com/paper/1908.10267/) (2019)

## Built on by (most cited first, 30 of 35)

- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [Flow Matching in Latent Space](https://mlsanity.com/paper/2307.08698/) (2023)
- [Deep Learning-based Image and Video Inpainting: A Survey](https://mlsanity.com/paper/2401.03395/) (2024)
- [MPIB: An MPI-Based Bokeh Rendering Framework for Realistic Partial Occlusion Effects](https://mlsanity.com/paper/2207.08403/) (2022)
- [Image Completion with Heterogeneously Filtered Spectral Hints](https://mlsanity.com/paper/2211.03700/) (2022)
- [Gradpaint: Gradient-Guided Inpainting with Diffusion Models](https://mlsanity.com/paper/2309.09614/) (2023)
- [GLaMa: Joint Spatial and Frequency Loss for General Image Inpainting](https://mlsanity.com/paper/2205.07162/) (2022)
- [Robust 3D Garment Digitization from Monocular 2D Images for 3D Virtual Try-On Systems](https://mlsanity.com/paper/2111.15140/) (2021)
- [RGI: robust GAN-inversion for mask-free image inpainting and unsupervised pixel-wise anomaly detection](https://mlsanity.com/paper/2302.12464/) (2023)
- [Image Inpainting via Iteratively Decoupled Probabilistic Modeling](https://mlsanity.com/paper/2212.02963/) (2022)
- [Feature Refinement to Improve High Resolution Image Inpainting](https://mlsanity.com/paper/2206.13644/) (2022)
- [PixelHacker: Image Inpainting with Structural and Semantic Consistency](https://mlsanity.com/paper/2504.20438/) (2025)
- [Cloth2Tex: A Customized Cloth Texture Generation Pipeline for 3D Virtual Try-On](https://mlsanity.com/paper/2308.04288/) (2023)
- [ReGO: Reference-Guided Outpainting for Scenery Image](https://mlsanity.com/paper/2106.10601/) (2021)
- [Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid](https://mlsanity.com/paper/2112.04107/) (2021)
- [Locate, Assign, Refine: Taming Customized Promptable Image Inpainting](https://mlsanity.com/paper/2403.19534/) (2024)
- [Learning Position-Aware Implicit Neural Network for Real-World Face Inpainting](https://mlsanity.com/paper/2401.10537/) (2024)
- [Object Remover Performance Evaluation Methods using Class-wise Object Removal Images](https://mlsanity.com/paper/2404.11104/) (2024)
- [WavePaint: Resource-efficient Token-mixer for Self-supervised Inpainting](https://mlsanity.com/paper/2307.00407/) (2023)
- [Neural Scene Designer: Self-Styled Semantic Image Manipulation](https://mlsanity.com/paper/2509.01405/) (2025)
- [Thermal Infrared Image Inpainting via Edge-Aware Guidance](https://mlsanity.com/paper/2210.16000/) (2022)
- [Line Drawing Guided Progressive Inpainting of Mural Damage](https://mlsanity.com/paper/2211.06649/) (2022)
- [SR-CurvANN: Advancing 3D Surface Reconstruction through Curvature-Aware Neural Networks](https://mlsanity.com/paper/2407.17896/) (2024)
- [Binary Diffusion Probabilistic Model](https://mlsanity.com/paper/2501.13915/) (2025)
- [Auxiliary Loss Reweighting for Image Inpainting](https://mlsanity.com/paper/2111.07279/) (2021)
- [Wavelet Prior Attention Learning in Axial Inpainting Network](https://mlsanity.com/paper/2206.03113/) (2022)
- [Both Spatial and Frequency Cues Contribute to High-Fidelity Image Inpainting](https://mlsanity.com/paper/2307.07678/) (2023)
- [SafePaint: Anti-forensic Image Inpainting with Domain Adaptation](https://mlsanity.com/paper/2404.18136/) (2024)
- [Task-Decoupled Image Inpainting Framework for Class-specific Object Remover](https://mlsanity.com/paper/2410.02894/) (2024)

## Abstract (arXiv)

Inpainting arbitrary missing regions is challenging because learning valid features for various masked regions is nontrivial. Though U-shaped encoder-decoder frameworks have been witnessed to be successful, most of them share a common drawback of mask unawareness in feature extraction because all convolution windows (or regions), including those with various shapes of missing pixels, are treated equally and filtered with fixed learned kernels. To this end, we propose our novel mask-aware inpainting solution. Firstly, a Mask-Aware Dynamic Filtering (MADF) module is designed to effectively learn multi-scale features for missing regions in the encoding phase. Specifically, filters for each convolution window are generated from features of the corresponding region of the mask. The second fold of mask awareness is achieved by adopting Point-wise Normalization (PN) in our decoding phase, considering that statistical natures of features at masked points differentiate from those of unmasked points. The proposed PN can tackle this issue by dynamically assigning point-wise scaling factor and bias. Lastly, our model is designed to be an end-to-end cascaded refinement one. Supervision information such as reconstruction loss, perceptual loss and total variation loss is incrementally leveraged to boost the inpainting results from coarse to fine. Effectiveness of the proposed framework is validated both quantitatively and qualitatively via extensive experiments on three public datasets including Places2, CelebA and Paris StreetView.