# AURA : Automatic Mask Generator using Randomized Input Sampling for Object Removal

arXiv: https://arxiv.org/abs/2305.07857 · HTML: https://mlsanity.com/paper/2305.07857/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 65 references that are in the map.
- First posted on arXiv 13 May 2023.

## Builds on (most cited first, 30 of 37)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting](https://mlsanity.com/paper/2005.09704/) (2020)
- [MSeg: A Composite Dataset for Multi-domain Semantic Segmentation](https://mlsanity.com/paper/2112.13762/) (2021)
- [Recurrent Feature Reasoning for Image Inpainting](https://mlsanity.com/paper/2008.03737/) (2020)
- [High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis](https://mlsanity.com/paper/1611.09969/) (2016)
- [Aggregated Contextual Transformations for High-Resolution Image Inpainting](https://mlsanity.com/paper/2104.01431/) (2021)
- [Shift-Net: Image Inpainting via Deep Feature Rearrangement](https://mlsanity.com/paper/1801.09392/) (2018)
- [Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding](https://mlsanity.com/paper/2203.00867/) (2022)
- [PD-GAN: Probabilistic Diverse GAN for Image Inpainting](https://mlsanity.com/paper/2105.02201/) (2021)
- [High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling](https://mlsanity.com/paper/2005.11742/) (2020)
- [TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations](https://mlsanity.com/paper/2103.15982/) (2021)
- [WaveFill: A Wavelet-based Generation Network for Image Inpainting](https://mlsanity.com/paper/2107.11027/) (2021)
- [WDNet: Watermark-Decomposition Network for Visible Watermark Removal](https://mlsanity.com/paper/2012.07616/) (2020)
- [Image Inpainting Guided by Coherence Priors of Semantics and Textures](https://mlsanity.com/paper/2012.08054/) (2020)
- [Reduce Information Loss in Transformers for Pluralistic Image Inpainting](https://mlsanity.com/paper/2205.05076/) (2022)

## Abstract (arXiv)

The objective of the image inpainting task is to fill missing regions of an image in a visually plausible way. Recently, deep-learning-based image inpainting networks have generated outstanding results, and some utilize their models as object removers by masking unwanted objects in an image. However, while trying to better remove objects using their networks, the previous works pay less attention to the importance of the input mask. In this paper, we focus on generating the input mask to better remove objects using the off-the-shelf image inpainting network. We propose an automatic mask generator inspired by the explainable AI (XAI) method, whose output can better remove objects than a semantic segmentation mask. The proposed method generates an importance map using randomly sampled input masks and quantitatively estimated scores of the completed images obtained from the random masks. The output mask is selected by a judge module among the candidate masks which are generated from the importance map. We design the judge module to quantitatively estimate the quality of the object removal results. In addition, we empirically find that the evaluation methods used in the previous works reporting object removal results are not appropriate for estimating the performance of an object remover. Therefore, we propose new evaluation metrics (FID^* and U-IDS^*) to properly evaluate the quality of object removers. Experiments confirm that our method shows better performance in removing target class objects than the masks generated from the semantic segmentation maps, and the two proposed metrics make judgments consistent with humans.