# Aggregated Contextual Transformations for High-Resolution Image Inpainting

arXiv: https://arxiv.org/abs/2104.01431 · HTML: https://mlsanity.com/paper/2104.01431/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 75 references that are in the map.
- First posted on arXiv 3 Apr 2021.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021)
- [Understanding Convolution for Semantic Segmentation](https://mlsanity.com/paper/1702.08502/) (2017)
- [EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning](https://mlsanity.com/paper/1901.00212/) (2019)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [Semantic Image Inpainting with Deep Generative Models](https://mlsanity.com/paper/1607.07539/) (2016)
- [High-Fidelity Pluralistic Image Completion with Transformers](https://mlsanity.com/paper/2103.14031/) (2021)
- [Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting](https://mlsanity.com/paper/2005.09704/) (2020)
- [StructureFlow: Image Inpainting via Structure-aware Appearance Flow](https://mlsanity.com/paper/1908.03852/) (2019)
- [Coherent Semantic Attention for Image Inpainting](https://mlsanity.com/paper/1905.12384/) (2019)
- [Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting](https://mlsanity.com/paper/1904.07475/) (2019)
- [High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis](https://mlsanity.com/paper/1611.09969/) (2016)

## Built on by (most cited first, 30 of 89)

- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Improving Diffusion Models for Inverse Problems using Manifold Constraints](https://mlsanity.com/paper/2206.00941/) (2022)
- [DiffIR: Efficient Diffusion Model for Image Restoration](https://mlsanity.com/paper/2303.09472/) (2023)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting](https://mlsanity.com/paper/2312.03594/) (2023)
- [VmambaIR: Visual State Space Model for Image Restoration](https://mlsanity.com/paper/2403.11423/) (2024)
- [Monte Carlo guided Diffusion for Bayesian linear inverse problems](https://mlsanity.com/paper/2308.07983/) (2023)
- [CoordFill: Efficient High-Resolution Image Inpainting via Parameterized Coordinate Querying](https://mlsanity.com/paper/2303.08524/) (2023)
- [Improving Diffusion-Based Image Synthesis with Context Prediction](https://mlsanity.com/paper/2401.02015/) (2024)
- [Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand](https://mlsanity.com/paper/2208.03382/) (2022)
- [FastScene: Text-Driven Fast 3D Indoor Scene Generation via Panoramic Gaussian Splatting](https://mlsanity.com/paper/2405.05768/) (2024)
- [Denoising Diffusion Models for 3D Healthy Brain Tissue Inpainting](https://mlsanity.com/paper/2403.14499/) (2024)
- [ZITS++: Image Inpainting by Improving the Incremental Transformer on Structural Priors](https://mlsanity.com/paper/2210.05950/) (2022)
- [Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency](https://mlsanity.com/paper/2312.04831/) (2023)
- [Reversing the Abnormal: Pseudo-Healthy Generative Networks for Anomaly Detection](https://mlsanity.com/paper/2303.08452/) (2023)
- [Image Completion with Heterogeneously Filtered Spectral Hints](https://mlsanity.com/paper/2211.03700/) (2022)
- [TextIR: A Simple Framework for Text-based Editable Image Restoration](https://mlsanity.com/paper/2302.14736/) (2023)
- [Deep Equilibrium Diffusion Restoration with Parallel Sampling](https://mlsanity.com/paper/2311.11600/) (2023)
- [Learning Prior Feature and Attention Enhanced Image Inpainting](https://mlsanity.com/paper/2208.01837/) (2022)
- [GLaMa: Joint Spatial and Frequency Loss for General Image Inpainting](https://mlsanity.com/paper/2205.07162/) (2022)
- [High-Fidelity Image Inpainting with GAN Inversion](https://mlsanity.com/paper/2208.11850/) (2022)
- [TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpainting](https://mlsanity.com/paper/2504.00996/) (2025)
- [An I2I Inpainting Approach for Efficient Channel Knowledge Map Construction](https://mlsanity.com/paper/2406.09822/) (2024)
- [Unbiased Multi-Modality Guidance for Image Inpainting](https://mlsanity.com/paper/2208.11844/) (2022)
- [Image Inpainting via Iteratively Decoupled Probabilistic Modeling](https://mlsanity.com/paper/2212.02963/) (2022)
- [NTIRE 2024 Restore Any Image Model (RAIM) in the Wild Challenge](https://mlsanity.com/paper/2405.09923/) (2024)
- [Feature Refinement to Improve High Resolution Image Inpainting](https://mlsanity.com/paper/2206.13644/) (2022)
- [TransCAB: Transferable Clean-Annotation Backdoor to Object Detection with Natural Trigger in Real-World](https://mlsanity.com/paper/2209.02339/) (2022)
- [Semantically Consistent Video Inpainting with Conditional Diffusion Models](https://mlsanity.com/paper/2405.00251/) (2024)

## Abstract (arXiv)

State-of-the-art image inpainting approaches can suffer from generating distorted structures and blurry textures in high-resolution images (e.g., 512x512). The challenges mainly drive from (1) image content reasoning from distant contexts, and (2) fine-grained texture synthesis for a large missing region. To overcome these two challenges, we propose an enhanced GAN-based model, named Aggregated COntextual-Transformation GAN (AOT-GAN), for high-resolution image inpainting. Specifically, to enhance context reasoning, we construct the generator of AOT-GAN by stacking multiple layers of a proposed AOT block. The AOT blocks aggregate contextual transformations from various receptive fields, allowing to capture both informative distant image contexts and rich patterns of interest for context reasoning. For improving texture synthesis, we enhance the discriminator of AOT-GAN by training it with a tailored mask-prediction task. Such a training objective forces the discriminator to distinguish the detailed appearances of real and synthesized patches, and in turn, facilitates the generator to synthesize clear textures. Extensive comparisons on Places2, the most challenging benchmark with 1.8 million high-resolution images of 365 complex scenes, show that our model outperforms the state-of-the-art by a significant margin in terms of FID with 38.60% relative improvement. A user study including more than 30 subjects further validates the superiority of AOT-GAN. We further evaluate the proposed AOT-GAN in practical applications, e.g., logo removal, face editing, and object removal. Results show that our model achieves promising completions in the real world. We release code and models in https://github.com/researchmm/AOT-GAN-for-Inpainting.