mlsanityData to 7 Oct 2026

Learning Image Priors through Patch-based Diffusion Models for Solving Inverse Problems

arXiv 2406.02462 ↗First posted 4 Jun 2024cs.CV cs.AIcited by 18

On the map: Diffusion models → Inverse problems and image restoration

Abstract (arXiv)

Diffusion models can learn strong image priors from underlying data distribution and use them to solve inverse problems, but the training process is computationally expensive and requires lots of data. Such bottlenecks prevent most existing works from being feasible for high-dimensional and high-resolution data such as 3D images. This paper proposes a method to learn an efficient data prior for the entire image by training diffusion models only on patches of images. Specifically, we propose a patch-based position-aware diffusion inverse solver, called PaDIS, where we obtain the score function of the whole image through scores of patches and their positional encoding and utilize this as the prior for solving inverse problems. First of all, we show that this diffusion model achieves an improved memory efficiency and data efficiency while still maintaining the capability to generate entire images via positional encoding. Additionally, the proposed PaDIS model is highly flexible and can be plugged in with different diffusion inverse solvers (DIS). We demonstrate that the proposed PaDIS approach enables solving various inverse problems in both natural and medical image domains, including CT reconstruction, deblurring, and superresolution, given only patch-based priors. Notably, PaDIS outperforms previous DIS methods trained on entire image priors in the case of limited training data, demonstrating the data efficiency of our proposed approach by learning patch-based prior.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models. Builds on 48 of its 67 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

48 of 67 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Monte Carlo guided Diffusion for Bayesian linear inverse problems15 Aug
  2. 2023Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency16 Jul
  3. 2023Direct Diffusion Bridge using Data Consistency for Inverse Problems31 May
  4. 2023Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration20 Mar
  5. 2022Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model1 Dec
  6. 2022Diffusion Posterior Sampling for General Noisy Inverse Problems29 Sep
  7. 2022Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Models29 Jul
  8. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraints2 Jun
  9. 2022Denoising Diffusion Restoration Models27 Jan
  10. 2021Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction9 Dec

Other references · 38

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  11. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  12. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
26 more
  1. 2021Solving Inverse Problems in Medical Imaging with Score-Based Generative Modelscited by 514
  2. 2016Deep Convolutional Neural Network for Inverse Problems in Imagingcited by 431
  3. 2021Score-based diffusion models for accelerated MRIcited by 308
  4. 2023I2SB: Image-to-Image Schr"odinger Bridgecited by 214
  5. 2021SNIPS: Solving Noisy Inverse Problems Stochasticallycited by 191
  6. 2023Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Modelscited by 131
  7. 2022Solving 3D Inverse Problems using Pre-trained 2D Diffusion Modelscited by 130
  8. 2023ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Modelscited by 127
  9. 2023Matryoshka Diffusion Modelscited by 70
  10. 2018Image Restoration using Total Variation Regularized Deep Image Priorcited by 65
  11. 2023Improving 3D Imaging with Pre-Trained Perpendicular 2D Diffusion Modelscited by 53
  12. 2020Scalable Plug-and-Play ADMM with Convergence Guaranteescited by 49
  13. 2023Patched Denoising Diffusion Models For High-Resolution Image Synthesiscited by 45
  14. 2020Provable Convergence of Plug-and-Play Priors with MMSE denoiserscited by 39
  15. 2022Online Deep Equilibrium Learning for Regularization by Denoisingcited by 22
  16. 2014Optimal low-rank approximations of Bayesian linear inverse problemscited by 21
  17. 2022Sparse-view Cone Beam CT Reconstruction using Data-consistent Supervised and Adversarial Learning from Scarce Training Datacited by 17
  18. 2022Patch-Based Denoising Diffusion Probabilistic Model for Sparse-View CT Reconstructioncited by 16
  19. 2022Improving Diffusion Model Efficiency Through Patchingcited by 15
  20. 2020Boosting the Performance of Plug-and-Play Priors via Denoiser Scalingcited by 12
  21. 2023A Complex Quasi-Newton Proximal Method for Image Reconstruction in Compressed Sensing MRIcited by 8
  22. 2021How Shift Equivariance Impacts Metric Learning for Instance Segmentationcited by 7
  23. 2023Parallel Diffusion Model-based Sparse-view Cone-beam Breast CTcited by 7
  24. 2023Constrained Regularization by Denoising with Automatic Parameter Selectioncited by 6
  25. 2022CoRRECT: A Deep Unfolding Framework for Motion-Corrected Quantitative R2* Mappingcited by 5
  26. 2022Monotonically Convergent Regularization by Denoisingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 10; 2026: 5.

18 papers.

Sort:

By area:

  1. 2025Diffusion models for inverse problemscited by 15
  2. 2025Learning Regularization Functionals for Inverse Problems: A Comparative Studycited by 12
  3. 2024Patch-Based Diffusion Models Beat Whole-Image Models for Mismatched Distribution Inverse Problemscited by 8
  4. 2024Particle-Filtering-based Latent Diffusion for Inverse Problemscited by 5
  5. 2026Improving 2D Diffusion Models for 3D Medical Imaging with Inter-Slice Consistent Stochasticitycited by 3
  6. 2025Investigating the Feasibility of Patch-based Inference for Generalized Diffusion Priors in Inverse Problems for Medical Imagescited by 2
  7. 2025Coupled Data and Measurement Space Dynamics for Enhanced Diffusion Posterior Samplingcited by 2
  8. 2024Filterless Snapshot Hyperspectral Imaging using Guided Patch Diffusioncited by 1
  9. 2025Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Datacited by 1
  10. 2025Local Patches Meet Global Context: Scalable 3D Diffusion Priors for Computed Tomography Reconstructioncited by 1
  11. 2025KernelFusion: Assumption-Free Blind Super-Resolution via Patch Diffusioncited by 0
  12. 2025Patch-Based Diffusion for Data-Efficient, Radiologist-Preferred MRI Reconstructioncited by 0
6 more
  1. 2025cryoSENSE: Compressive Sensing Enables High-throughput Microscopy with Sparse and Generative Priors on the Protein Cryo-EM Image Manifoldcited by 0
  2. 2025Towards High-Fidelity Gaussian Splatting with Queried-Convolution Neural Networkscited by 0
  3. 2026KLIP: localized distribution shift detection via KL-divergence with diffusion priors in Inverse Problemscited by 0
  4. 2026Patched Flow Matching: Generative Wall-Pressure Reconstruction Beyond Training-Domain Scales from Sparse Sensorscited by 0
  5. 2026Spatio-Temporal Mixture-of-Modality-Experts Diffusion for Quantitative DCE-MRI Synthesis from Incomplete MR Sequencescited by 0
  6. 2026P-PatchDiff: Progressive Patch Diffusion Models for Low-light Image Enhancementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DiffusionBlend: Learning 3D Image Prior through Position-aware Diffusion Score Blending for 3D Computed Tomography Reconstruction
  2. 2026Backward SDEs-based Diffusion for Physics-Constrained Generation
  3. 2024A Survey on Diffusion Models for Inverse Problems
  4. 2025Inference-Time Search Using Side Information for Diffusion-Based Image Reconstruction
  5. 2025Diff-Unfolding: A Model-Based Score Learning Framework for Inverse Problems
  6. 2023A Variational Perspective on Solving Inverse Problems with Diffusion Models
  7. 2025Diffusion Active Learning: Towards Data-Driven Experimental Design in Computed Tomography
  8. 2024Deep Diffusion Image Prior for Efficient OOD Adaptation in 3D Inverse Problems