mlsanityData to 14 Oct 2026

Patch-Based Diffusion Models Beat Whole-Image Models for Mismatched Distribution Inverse Problems

arXiv 2410.11730 ↗First posted 15 Oct 2024cs.CV cs.AI eess.IVcited by 8

On the map: Diffusion models → Inverse problems and image restoration

Abstract (arXiv)

Diffusion models have achieved excellent success in solving inverse problems due to their ability to learn strong image priors, but existing approaches require a large training dataset of images that should come from the same distribution as the test dataset. When the training and test distributions are mismatched, artifacts and hallucinations can occur in reconstructed images due to the incorrect priors. In this work, we systematically study out of distribution (OOD) problems where a known training distribution is first provided. We first study the setting where only a single measurement obtained from the unknown test distribution is available. Next we study the setting where a very small sample of data belonging to the test distribution is available, and our goal is still to reconstruct an image from a measurement that came from the test distribution. In both settings, we use a patch-based diffusion prior that learns the image distribution solely from patches. Furthermore, in the first setting, we include a self-supervised loss that helps the network output maintain consistency with the measurement. Extensive experiments show that in both settings, the patch-based method can obtain high quality image reconstructions that can outperform whole-image models and can compete with methods that have access to large in-distribution training datasets. Furthermore, we show how whole-image models are prone to memorization and overfitting, leading to artifacts in the reconstructions, while a patch-based model can resolve these issues.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models. Builds on 39 of its 55 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

39 of 55 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Deep Diffusion Image Prior for Efficient OOD Adaptation in 3D Inverse Problems15 Jul
  2. 2024Learning Image Priors through Patch-based Diffusion Models for Solving Inverse Problems4 Jun
  3. 2024Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariance3 Feb
  4. 2023Direct Diffusion Bridge using Data Consistency for Inverse Problems31 May
  5. 2023Poisson-Gaussian Holographic Phase Retrieval with Score-based Image Prior12 May
  6. 2023Score-Based Diffusion Models as Principled Priors for Inverse Imaging23 Apr
  7. 2023Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problems10 Mar
  8. 2022Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model1 Dec
  9. 2022Diffusion Posterior Sampling for General Noisy Inverse Problems29 Sep
  10. 2022Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Models29 Jul
  11. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraints2 Jun
  12. 2022Denoising Diffusion Restoration Models27 Jan

Other references · 27

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  6. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  7. 2017Deep Image Priorcited by 1,098
  8. 2021Solving Inverse Problems in Medical Imaging with Score-Based Generative Modelscited by 521
  9. 2021Robust Compressed Sensing MRI with Deep Generative Priorscited by 286
  10. 2019Plug-and-Play Methods Provably Converge with Properly Trained Denoiserscited by 198
  11. 2023Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Modelscited by 133
  12. 2015Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolationcited by 78
15 more
  1. 2021Dynamic CT Reconstruction from Limited Views with Implicit Neural Representations and Parametric Motion Fieldscited by 71
  2. 2018Image Restoration using Total Variation Regularized Deep Image Priorcited by 65
  3. 2020Scalable Plug-and-Play ADMM with Convergence Guaranteescited by 50
  4. 2023Patched Denoising Diffusion Models For High-Resolution Image Synthesiscited by 45
  5. 2021Rethinking Deep Image Prior for Denoisingcited by 38
  6. 2021An Educated Warm Start For Deep Image Prior-Based Micro CT Reconstructioncited by 16
  7. 2023DomainStudio: Fine-Tuning Diffusion Models for Domain-Driven Image Generation using Limited Datacited by 15
  8. 2024DiffusionBlend: Learning 3D Image Prior through Position-aware Diffusion Score Blending for 3D Computed Tomography Reconstructioncited by 13
  9. 2019Online Regularization by Denoising with Applications to Phase Retrievalcited by 12
  10. 2020Boosting the Performance of Plug-and-Play Priors via Denoiser Scalingcited by 12
  11. 2024Neural Approximate Mirror Maps for Constrained Diffusion Modelscited by 9
  12. 2023A Complex Quasi-Newton Proximal Method for Image Reconstruction in Compressed Sensing MRIcited by 8
  13. 2024Provable Preconditioned Plug-and-Play Approach for Compressed Sensing MRI Reconstructioncited by 5
  14. 2022Monotonically Convergent Regularization by Denoisingcited by 3
  15. 2019Solving RED with Weighted Proximal Methodscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 1.

8 papers.

  1. 2024Likelihood-Scheduled Score-Based Generative Modeling for Fully 3D PET Image Reconstructioncited by 9
  2. 2025Investigating the Feasibility of Patch-based Inference for Generalized Diffusion Priors in Inverse Problems for Medical Imagescited by 2
  3. 2025Personalized MR-Informed Diffusion Models for 3D PET Image Reconstructioncited by 2
  4. 2024Filterless Snapshot Hyperspectral Imaging using Guided Patch Diffusioncited by 1
  5. 2025Supervised Diffusion-Model-Based PET Image Reconstructioncited by 1
  6. 2026Composing diffusion priors with explicit physical context via generative Gibbs samplingcited by 1
  7. 2025KernelFusion: Assumption-Free Blind Super-Resolution via Patch Diffusioncited by 0
  8. 2025Patch-Based Diffusion for Data-Efficient, Radiologist-Preferred MRI Reconstructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Diffusion Models for Inverse Problems
  2. 2026Efficient and Training-Free Single-Image Diffusion Models
  3. 2023Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency
  4. 2024Learning Diffusion Model from Noisy Measurement using Principled Expectation-Maximization Method
  5. 2025Diffusion models for inverse problems
  6. 2022Improving Diffusion Model Efficiency Through Patching
  7. 2025APT: Improving Diffusion Models for High Resolution Image Generation with Adaptive Path Tracing
  8. 2023Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models