# Learning Sparse Masks for Diffusion-based Image Inpainting

arXiv: https://arxiv.org/abs/2110.02636 · HTML: https://mlsanity.com/paper/2110.02636/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 6 Oct 2021.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Coherent Semantic Attention for Image Inpainting](https://mlsanity.com/paper/1905.12384/) (2019)
- [High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis](https://mlsanity.com/paper/1611.09969/) (2016)
- [iPiano: Inertial Proximal Algorithm for Non-Convex Optimization](https://mlsanity.com/paper/1404.4805/) (2014)
- [Connections between Numerical Algorithms for PDEs and Neural Networks](https://mlsanity.com/paper/2107.14742/) (2021)
- [Adaptive Image Sampling using Deep Learning and its Application on X-Ray Fluorescence Image Reconstruction](https://mlsanity.com/paper/1812.10836/) (2018)
- [A bi-level view of inpainting - based image compression](https://mlsanity.com/paper/1401.4112/) (2014)
- [Efficient Data Optimisation for Harmonic Inpainting with Finite Elements](https://mlsanity.com/paper/2105.01586/) (2021)
- [Image Inpainting Using Wasserstein Generative Adversarial Imputation Network](https://mlsanity.com/paper/2106.15341/) (2021)
- [A Systematic Evaluation of Coding Strategies for Sparse Binary Images](https://mlsanity.com/paper/2010.13634/) (2020)
- [Sparse Inpainting with Smoothed Particle Hydrodynamics](https://mlsanity.com/paper/2011.11289/) (2020)
- [Optimising Spatial and Tonal Data for PDE-based Inpainting](https://mlsanity.com/paper/1506.04566/) (2015)
- [Inpainting-based Video Compression in FullHD](https://mlsanity.com/paper/2008.10273/) (2020)
- [Representing Images in 200 Bytes: Compression via Triangulation](https://mlsanity.com/paper/1809.02257/) (2018)

## Built on by (most cited first, 5 of 5)

- [Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyond](https://mlsanity.com/paper/2304.04968/) (2023)
- [Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?](https://mlsanity.com/paper/2305.12954/) (2023)
- [ToddlerDiffusion: Interactive Structured Image Generation with Cascaded Schr"odinger Bridge](https://mlsanity.com/paper/2311.14542/) (2023)
- [Efficient Neural Generation of 4K Masks for Homogeneous Diffusion Inpainting](https://mlsanity.com/paper/2303.10096/) (2023)
- [A Wasserstein GAN for Joint Learning of Inpainting and Spatial Optimisation](https://mlsanity.com/paper/2202.05623/) (2022)

## Abstract (arXiv)

Diffusion-based inpainting is a powerful tool for the reconstruction of images from sparse data. Its quality strongly depends on the choice of known data. Optimising their spatial location -- the inpainting mask -- is challenging. A commonly used tool for this task are stochastic optimisation strategies. However, they are slow as they compute multiple inpainting results. We provide a remedy in terms of a learned mask generation model. By emulating the complete inpainting pipeline with two networks for mask generation and neural surrogate inpainting, we obtain a model for highly efficient adaptive mask generation. Experiments indicate that our model can achieve competitive quality with an acceleration by as much as four orders of magnitude. Our findings serve as a basis for making diffusion-based inpainting more attractive for applications such as image compression, where fast encoding is highly desirable.