# From Posterior Sampling to Meaningful Diversity in Image Restoration

arXiv: https://arxiv.org/abs/2310.16047 · HTML: https://mlsanity.com/paper/2310.16047/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 63 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 30 of 49)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better](https://mlsanity.com/paper/1908.03826/) (2019)
- [Deep Back-Projection Networks For Super-Resolution](https://mlsanity.com/paper/1803.02735/) (2018)
- [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [SRFlow: Learning the Super-Resolution Space with Normalizing Flow](https://mlsanity.com/paper/2006.14200/) (2020)
- [SNIPS: Solving Noisy Inverse Problems Stochastically](https://mlsanity.com/paper/2105.14951/) (2021)
- [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019)
- [Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model](https://mlsanity.com/paper/1904.09770/) (2019)
- [Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis](https://mlsanity.com/paper/1903.05628/) (2019)
- [High-Fidelity Pluralistic Image Completion with Transformers](https://mlsanity.com/paper/2103.14031/) (2021)
- [PartImageNet: A Large, High-Quality Dataset of Parts](https://mlsanity.com/paper/2112.00933/) (2021)
- [Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging](https://mlsanity.com/paper/2202.05265/) (2022)
- [Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE](https://mlsanity.com/paper/2103.10022/) (2021)
- [PD-GAN: Probabilistic Diverse GAN for Image Inpainting](https://mlsanity.com/paper/2105.02201/) (2021)
- [Generating High Fidelity Data from Low-density Regions using Diffusion Models](https://mlsanity.com/paper/2203.17260/) (2022)

## Built on by (most cited first, 12 of 12)

- [Compressed Image Generation with Denoising Diffusion Codebook Models](https://mlsanity.com/paper/2502.01189/) (2025)
- [Rethinking Diffusion Posterior Sampling: From Conditional Score Estimator to Maximizing a Posterior](https://mlsanity.com/paper/2501.18913/) (2025)
- [PFStorer: Personalized Face Restoration and Super-Resolution](https://mlsanity.com/paper/2403.08436/) (2024)
- [Segmentation-guided MRI reconstruction for meaningfully diverse reconstructions](https://mlsanity.com/paper/2407.18026/) (2024)
- [I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting](https://mlsanity.com/paper/2411.19050/) (2024)
- [Hierarchical Uncertainty Exploration via Feedforward Posterior Trees](https://mlsanity.com/paper/2405.15719/) (2024)
- [Uncertainty Visualization via Low-Dimensional Posterior Projections](https://mlsanity.com/paper/2312.07804/) (2023)
- [On-the-fly Repulsion in the Contextual Space for Rich Diversity in Diffusion Transformers](https://mlsanity.com/paper/2603.28762/) (2026)
- [Mind the Detail: Uncovering Clinically Relevant Image Details in Accelerated MRI with Semantically Diverse Reconstructions](https://mlsanity.com/paper/2507.00670/) (2025)
- [Robustness and Exploration of Variational and Machine Learning Approaches to Inverse Problems: An Overview](https://mlsanity.com/paper/2402.12072/) (2024)
- [SP3: Spherical Priors for Plug-and-Play Restoration](https://mlsanity.com/paper/2606.16396/) (2026)
- [Semantic Browsing: Controllable Diversity for Image Generation](https://mlsanity.com/paper/2606.23679/) (2026)

## Abstract (arXiv)

Image restoration problems are typically ill-posed in the sense that each degraded image can be restored in infinitely many valid ways. To accommodate this, many works generate a diverse set of outputs by attempting to randomly sample from the posterior distribution of natural images given the degraded input. Here we argue that this strategy is commonly of limited practical value because of the heavy tail of the posterior distribution. Consider for example inpainting a missing region of the sky in an image. Since there is a high probability that the missing region contains no object but clouds, any set of samples from the posterior would be entirely dominated by (practically identical) completions of sky. However, arguably, presenting users with only one clear sky completion, along with several alternative solutions such as airships, birds, and balloons, would better outline the set of possibilities. In this paper, we initiate the study of meaningfully diverse image restoration. We explore several post-processing approaches that can be combined with any diverse image restoration method to yield semantically meaningful diversity. Moreover, we propose a practical approach for allowing diffusion based image restoration methods to generate meaningfully diverse outputs, while incurring only negligent computational overhead. We conduct extensive user studies to analyze the proposed techniques, and find the strategy of reducing similarity between outputs to be significantly favorable over posterior sampling. Code and examples are available at https://noa-cohen.github.io/MeaningfulDiversityInIR.