# Learning Image Priors through Patch-based Diffusion Models for Solving Inverse Problems

arXiv: https://arxiv.org/abs/2406.02462 · HTML: https://mlsanity.com/paper/2406.02462/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 48 of its 67 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 30 of 48)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [Solving Inverse Problems in Medical Imaging with Score-Based Generative Models](https://mlsanity.com/paper/2111.08005/) (2021)
- [Improving Diffusion Models for Inverse Problems using Manifold Constraints](https://mlsanity.com/paper/2206.00941/) (2022)
- [Deep Convolutional Neural Network for Inverse Problems in Imaging](https://mlsanity.com/paper/1611.03679/) (2016)
- [Score-based diffusion models for accelerated MRI](https://mlsanity.com/paper/2110.05243/) (2021)
- [Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction](https://mlsanity.com/paper/2112.05146/) (2021)
- [I2SB: Image-to-Image Schr"odinger Bridge](https://mlsanity.com/paper/2302.05872/) (2023)
- [Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Models](https://mlsanity.com/paper/2207.14626/) (2022)
- [SNIPS: Solving Noisy Inverse Problems Stochastically](https://mlsanity.com/paper/2105.14951/) (2021)
- [Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency](https://mlsanity.com/paper/2307.08123/) (2023)
- [Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration](https://mlsanity.com/paper/2303.11435/) (2023)
- [Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models](https://mlsanity.com/paper/2304.12526/) (2023)
- [Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2211.10655/) (2022)
- [ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models](https://mlsanity.com/paper/2310.07702/) (2023)
- [Matryoshka Diffusion Models](https://mlsanity.com/paper/2310.15111/) (2023)
- [Direct Diffusion Bridge using Data Consistency for Inverse Problems](https://mlsanity.com/paper/2305.19809/) (2023)

## Built on by (most cited first, 20 of 20)

- [Diffusion models for inverse problems](https://mlsanity.com/paper/2508.01975/) (2025)
- [Learning Regularization Functionals for Inverse Problems: A Comparative Study](https://mlsanity.com/paper/2510.01755/) (2025)
- [Patch-Based Diffusion Models Beat Whole-Image Models for Mismatched Distribution Inverse Problems](https://mlsanity.com/paper/2410.11730/) (2024)
- [Particle-Filtering-based Latent Diffusion for Inverse Problems](https://mlsanity.com/paper/2408.13868/) (2024)
- [Improving 2D Diffusion Models for 3D Medical Imaging with Inter-Slice Consistent Stochasticity](https://mlsanity.com/paper/2602.04162/) (2026)
- [Investigating the Feasibility of Patch-based Inference for Generalized Diffusion Priors in Inverse Problems for Medical Images](https://mlsanity.com/paper/2501.15309/) (2025)
- [Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Data](https://mlsanity.com/paper/2509.00924/) (2025)
- [Coupled Data and Measurement Space Dynamics for Enhanced Diffusion Posterior Sampling](https://mlsanity.com/paper/2510.09676/) (2025)
- [Filterless Snapshot Hyperspectral Imaging using Guided Patch Diffusion](https://mlsanity.com/paper/2412.02798/) (2024)
- [Local Patches Meet Global Context: Scalable 3D Diffusion Priors for Computed Tomography Reconstruction](https://mlsanity.com/paper/2512.18161/) (2025)
- [KernelFusion: Assumption-Free Blind Super-Resolution via Patch Diffusion](https://mlsanity.com/paper/2503.21907/) (2025)
- [Patch-Based Diffusion for Data-Efficient, Radiologist-Preferred MRI Reconstruction](https://mlsanity.com/paper/2509.21531/) (2025)
- [cryoSENSE: Compressive Sensing Enables High-throughput Microscopy with Sparse and Generative Priors on the Protein Cryo-EM Image Manifold](https://mlsanity.com/paper/2511.12931/) (2025)
- [Towards High-Fidelity Gaussian Splatting with Queried-Convolution Neural Networks](https://mlsanity.com/paper/2512.12898/) (2025)
- [KLIP: localized distribution shift detection via KL-divergence with diffusion priors in Inverse Problems](https://mlsanity.com/paper/2605.31596/) (2026)
- [Patched Flow Matching: Generative Wall-Pressure Reconstruction Beyond Training-Domain Scales from Sparse Sensors](https://mlsanity.com/paper/2606.22084/) (2026)
- [Spatio-Temporal Mixture-of-Modality-Experts Diffusion for Quantitative DCE-MRI Synthesis from Incomplete MR Sequences](https://mlsanity.com/paper/2606.25535/) (2026)
- [P-PatchDiff: Progressive Patch Diffusion Models for Low-light Image Enhancement](https://mlsanity.com/paper/2609.01123/) (2026)
- [One Evaluation, Any Operating Point: Hypernetwork-Amortized MeanFlow for 3D MRI Reconstruction](https://mlsanity.com/paper/2609.31655/) (2026)
- [Learning Field Reconstruction from Incomplete Data by Globally Correcting Local Estimates](https://mlsanity.com/paper/2610.05375/) (2026)

## Abstract (arXiv)

Diffusion models can learn strong image priors from underlying data distribution and use them to solve inverse problems, but the training process is computationally expensive and requires lots of data. Such bottlenecks prevent most existing works from being feasible for high-dimensional and high-resolution data such as 3D images. This paper proposes a method to learn an efficient data prior for the entire image by training diffusion models only on patches of images. Specifically, we propose a patch-based position-aware diffusion inverse solver, called PaDIS, where we obtain the score function of the whole image through scores of patches and their positional encoding and utilize this as the prior for solving inverse problems. First of all, we show that this diffusion model achieves an improved memory efficiency and data efficiency while still maintaining the capability to generate entire images via positional encoding. Additionally, the proposed PaDIS model is highly flexible and can be plugged in with different diffusion inverse solvers (DIS). We demonstrate that the proposed PaDIS approach enables solving various inverse problems in both natural and medical image domains, including CT reconstruction, deblurring, and superresolution, given only patch-based priors. Notably, PaDIS outperforms previous DIS methods trained on entire image priors in the case of limited training data, demonstrating the data efficiency of our proposed approach by learning patch-based prior.