mlsanityData to 7 Oct 2026

Bigger Isn't Always Better: Towards a General Prior for Medical Image Reconstruction

arXiv 2501.07376 ↗First posted 13 Jan 2025eess.IVcited by 2

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

Diffusion model have been successfully applied to many inverse problems, including MRI and CT reconstruction. Researchers typically re-purpose models originally designed for unconditional sampling without modifications. Using two different posterior sampling algorithms, we show empirically that such large networks are not necessary. Our smallest model, effectively a ResNet, performs almost as good as an attention U-Net on in-distribution reconstruction, while being significantly more robust towards distribution shifts. Furthermore, we introduce models trained on natural images and demonstrate that they can be used in both MRI and CT reconstruction, out-performing model trained on medical images in out-of-distribution cases. As a result of our findings, we strongly caution against simply re-using very large networks and encourage researchers to adapt the model complexity to the respective task. Moreover, we argue that a key step towards a general diffusion-based prior is training on natural images.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 54 of its 84 references that are in the map. First posted on arXiv 13 Jan 2025.

Builds on

54 of 84 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Improving 3D Imaging with Pre-Trained Perpendicular 2D Diffusion Models15 Mar
  2. 2023Patched Diffusion Models for Unsupervised Anomaly Detection in Brain MRI7 Mar
  3. 2021Solving Inverse Problems in Medical Imaging with Score-Based Generative Models15 Nov
  4. 2021Score-based diffusion models for accelerated MRI8 Oct

Other references · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  11. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
38 more
  1. 2017Non-local Neural Networkscited by 2,290
  2. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  3. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  4. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
  5. 2022Denoising Diffusion Restoration Modelscited by 951
  6. 2023Extracting Training Data from Diffusion Modelscited by 723
  7. 2017Exploring Generalization in Deep Learningcited by 595
  8. 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 549
  9. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  10. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  11. 2017MoDL: Model Based Deep Learning Architecture for Inverse Problemscited by 384
  12. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 381
  13. 2017Learning a Variational Network for Reconstruction of Accelerated MRI Datacited by 340
  14. 2023Denoising Diffusion Models for Plug-and-Play Image Restorationcited by 335
  15. 2021Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contractioncited by 306
  16. 2021Robust Compressed Sensing MRI with Deep Generative Priorscited by 284
  17. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  18. 2023Implicit Diffusion Models for Continuous Super-Resolutioncited by 216
  19. 2022Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Modelscited by 215
  20. 2023Understanding and Mitigating Copying in Diffusion Modelscited by 205
  21. 2021SNIPS: Solving Noisy Inverse Problems Stochasticallycited by 191
  22. 2024Dynamical Regimes of Diffusion Modelscited by 133
  23. 2024An analytic theory of creativity in convolutional diffusion modelscited by 103
  24. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  25. 2017A Deep Cascade of Convolutional Neural Networks for MR Image Reconstructioncited by 74
  26. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  27. 2023Closed-Form Diffusion Modelscited by 46
  28. 2023Patched Denoising Diffusion Models For High-Resolution Image Synthesiscited by 45
  29. 2022Stable Deep MRI Reconstruction using Generative Priorscited by 21
  30. 2024Diffusion Posterior Sampling is Computationally Intractablecited by 20
  31. 2017What are the Receptive, Effective Receptive, and Projective Fields of Neurons in Convolutional Neural Networks?cited by 16
  32. 2023Neural-network-based regularization methods for inverse problems in imagingcited by 14
  33. 2023Robustness of Deep Learning for Accelerated MRI: Benefits of Diverse Training Datacited by 10
  34. 2022Computed Tomography Reconstruction using Generative Energy-Based Priorscited by 9
  35. 2023Product of Gaussian Mixture Diffusion Modelscited by 6
  36. 2024Glimpse: Generalized Locality for Scalable and Robust CTcited by 6
  37. 2023Generative Priors for MRI Reconstruction Trained from Magnitude-Only Images Using Phase Augmentationcited by 3
  38. 2024LoFi: Neural Local Fields for Scalable Image Reconstructioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Weak Diffusion Priors Can Still Achieve Strong Inverse-Problem Performancecited by 4
  2. 2026Product-of-Gaussian-Mixture Diffusion Models for Joint Nonlinear MRI Reconstructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Sequential Diffusion-Guided Deep Image Prior For Medical Image Reconstruction
  2. 2025When are Diffusion Priors Helpful in Sparse Reconstruction? A Study with Sparse-view CT
  3. 2025Deep generative priors for 3D brain analysis
  4. 2023Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency
  5. 2022Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models
  6. 2025Investigating the Feasibility of Patch-based Inference for Generalized Diffusion Priors in Inverse Problems for Medical Images
  7. 2026Scale-Cascaded Diffusion Models for Super-Resolution in Medical Imaging
  8. 2022Diffusion Models for Medical Image Analysis: A Comprehensive Survey