# Patch-Based Diffusion Models Beat Whole-Image Models for Mismatched Distribution Inverse Problems

arXiv: https://arxiv.org/abs/2410.11730 · HTML: https://mlsanity.com/paper/2410.11730/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 39 of its 55 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 30 of 39)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [Solving Inverse Problems in Medical Imaging with Score-Based Generative Models](https://mlsanity.com/paper/2111.08005/) (2021)
- [Improving Diffusion Models for Inverse Problems using Manifold Constraints](https://mlsanity.com/paper/2206.00941/) (2022)
- [Robust Compressed Sensing MRI with Deep Generative Priors](https://mlsanity.com/paper/2108.01368/) (2021)
- [Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Models](https://mlsanity.com/paper/2207.14626/) (2022)
- [Plug-and-Play Methods Provably Converge with Properly Trained Denoisers](https://mlsanity.com/paper/1905.05406/) (2019)
- [Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models](https://mlsanity.com/paper/2304.12526/) (2023)
- [Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problems](https://mlsanity.com/paper/2303.05754/) (2023)
- [Score-Based Diffusion Models as Principled Priors for Inverse Imaging](https://mlsanity.com/paper/2304.11751/) (2023)
- [Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation](https://mlsanity.com/paper/1512.07331/) (2015)
- [Dynamic CT Reconstruction from Limited Views with Implicit Neural Representations and Parametric Motion Fields](https://mlsanity.com/paper/2104.11745/) (2021)
- [Direct Diffusion Bridge using Data Consistency for Inverse Problems](https://mlsanity.com/paper/2305.19809/) (2023)
- [Image Restoration using Total Variation Regularized Deep Image Prior](https://mlsanity.com/paper/1810.12864/) (2018)
- [Scalable Plug-and-Play ADMM with Convergence Guarantees](https://mlsanity.com/paper/2006.03224/) (2020)
- [Patched Denoising Diffusion Models For High-Resolution Image Synthesis](https://mlsanity.com/paper/2308.01316/) (2023)
- [Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariance](https://mlsanity.com/paper/2402.02149/) (2024)
- [Rethinking Deep Image Prior for Denoising](https://mlsanity.com/paper/2108.12841/) (2021)
- [Deep Diffusion Image Prior for Efficient OOD Adaptation in 3D Inverse Problems](https://mlsanity.com/paper/2407.10641/) (2024)
- [Learning Image Priors through Patch-based Diffusion Models for Solving Inverse Problems](https://mlsanity.com/paper/2406.02462/) (2024)
- [An Educated Warm Start For Deep Image Prior-Based Micro CT Reconstruction](https://mlsanity.com/paper/2111.11926/) (2021)
- [DomainStudio: Fine-Tuning Diffusion Models for Domain-Driven Image Generation using Limited Data](https://mlsanity.com/paper/2306.14153/) (2023)

## Built on by (most cited first, 8 of 8)

- [Likelihood-Scheduled Score-Based Generative Modeling for Fully 3D PET Image Reconstruction](https://mlsanity.com/paper/2412.04339/) (2024)
- [Investigating the Feasibility of Patch-based Inference for Generalized Diffusion Priors in Inverse Problems for Medical Images](https://mlsanity.com/paper/2501.15309/) (2025)
- [Personalized MR-Informed Diffusion Models for 3D PET Image Reconstruction](https://mlsanity.com/paper/2506.03804/) (2025)
- [Filterless Snapshot Hyperspectral Imaging using Guided Patch Diffusion](https://mlsanity.com/paper/2412.02798/) (2024)
- [Supervised Diffusion-Model-Based PET Image Reconstruction](https://mlsanity.com/paper/2506.24034/) (2025)
- [Composing diffusion priors with explicit physical context via generative Gibbs sampling](https://mlsanity.com/paper/2605.10642/) (2026)
- [KernelFusion: Assumption-Free Blind Super-Resolution via Patch Diffusion](https://mlsanity.com/paper/2503.21907/) (2025)
- [Patch-Based Diffusion for Data-Efficient, Radiologist-Preferred MRI Reconstruction](https://mlsanity.com/paper/2509.21531/) (2025)

## Abstract (arXiv)

Diffusion models have achieved excellent success in solving inverse problems due to their ability to learn strong image priors, but existing approaches require a large training dataset of images that should come from the same distribution as the test dataset. When the training and test distributions are mismatched, artifacts and hallucinations can occur in reconstructed images due to the incorrect priors. In this work, we systematically study out of distribution (OOD) problems where a known training distribution is first provided. We first study the setting where only a single measurement obtained from the unknown test distribution is available. Next we study the setting where a very small sample of data belonging to the test distribution is available, and our goal is still to reconstruct an image from a measurement that came from the test distribution. In both settings, we use a patch-based diffusion prior that learns the image distribution solely from patches. Furthermore, in the first setting, we include a self-supervised loss that helps the network output maintain consistency with the measurement. Extensive experiments show that in both settings, the patch-based method can obtain high quality image reconstructions that can outperform whole-image models and can compete with methods that have access to large in-distribution training datasets. Furthermore, we show how whole-image models are prone to memorization and overfitting, leading to artifacts in the reconstructions, while a patch-based model can resolve these issues.