# Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariance

arXiv: https://arxiv.org/abs/2402.02149 · HTML: https://mlsanity.com/paper/2402.02149/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 35 of its 44 references that are in the map.
- First posted on arXiv 3 Feb 2024.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 30 of 35)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [Solving Inverse Problems in Medical Imaging with Score-Based Generative Models](https://mlsanity.com/paper/2111.08005/) (2021)
- [Tackling the Generative Learning Trilemma with Denoising Diffusion GANs](https://mlsanity.com/paper/2112.07804/) (2021)
- [Denoising Diffusion Models for Plug-and-Play Image Restoration](https://mlsanity.com/paper/2305.08995/) (2023)
- [Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.06503/) (2022)
- [Deep Unfolding Network for Image Super-Resolution](https://mlsanity.com/paper/2003.10428/) (2020)
- [Deblurring via Stochastic Refinement](https://mlsanity.com/paper/2112.02475/) (2021)
- [A Variational Perspective on Solving Inverse Problems with Diffusion Models](https://mlsanity.com/paper/2305.04391/) (2023)
- [Nonlinear Transform Coding](https://mlsanity.com/paper/2007.03034/) (2020)
- [Score-Based Diffusion Models as Principled Priors for Inverse Imaging](https://mlsanity.com/paper/2304.11751/) (2023)
- [Refusion: Enabling Large-Size Realistic Image Restoration with Latent-Space Diffusion Models](https://mlsanity.com/paper/2304.08291/) (2023)
- [Taming VAEs](https://mlsanity.com/paper/1810.00597/) (2018)
- [Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models](https://mlsanity.com/paper/2206.07309/) (2022)
- [Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matching](https://mlsanity.com/paper/2206.08265/) (2022)
- [Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion](https://mlsanity.com/paper/2312.00852/) (2023)
- [Training-free Linear Image Inverses via Flows](https://mlsanity.com/paper/2310.04432/) (2023)
- [Estimating High Order Gradients of the Data Distribution by Denoising](https://mlsanity.com/paper/2111.04726/) (2021)

## Built on by (most cited first, 30 of 44)

- [Taming Rectified Flow for Inversion and Editing](https://mlsanity.com/paper/2411.04746/) (2024)
- [Learning Diffusion Priors from Observations by Expectation Maximization](https://mlsanity.com/paper/2405.13712/) (2024)
- [Steering Rectified Flow Models in the Vector Field for Controlled Image Generation](https://mlsanity.com/paper/2412.00100/) (2024)
- [Ensemble Kalman Diffusion Guidance: A Derivative-free Method for Inverse Problems](https://mlsanity.com/paper/2409.20175/) (2024)
- [Diffusion models for inverse problems](https://mlsanity.com/paper/2508.01975/) (2025)
- [Diffusion State-Guided Projected Gradient for Inverse Problems](https://mlsanity.com/paper/2410.03463/) (2024)
- [Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costs](https://mlsanity.com/paper/2410.11149/) (2024)
- [IRBridge: Solving Image Restoration Bridge with Pre-trained Generative Diffusion Models](https://mlsanity.com/paper/2505.24406/) (2025)
- [Inverse Problems with Diffusion Models: A MAP Estimation Perspective](https://mlsanity.com/paper/2407.20784/) (2024)
- [Equivariant Denoisers Cannot Copy Graphs: Align Your Graph Diffusion Models](https://mlsanity.com/paper/2405.17656/) (2024)
- [Patch-Based Diffusion Models Beat Whole-Image Models for Mismatched Distribution Inverse Problems](https://mlsanity.com/paper/2410.11730/) (2024)
- [Amortized Posterior Sampling with Diffusion Prior Distillation](https://mlsanity.com/paper/2407.17907/) (2024)
- [Gaussian is All You Need: A Unified Framework for Solving Inverse Problems via Diffusion Posterior Sampling](https://mlsanity.com/paper/2409.08906/) (2024)
- [Enhancing Diffusion Model Stability for Image Restoration via Gradient Management](https://mlsanity.com/paper/2507.06656/) (2025)
- [Conditional Mutual Information Based Diffusion Posterior Sampling for Solving Inverse Problems](https://mlsanity.com/paper/2501.02880/) (2025)
- [EquiReg: Equivariance Regularized Diffusion for Inverse Problems](https://mlsanity.com/paper/2505.22973/) (2025)
- [PriorGuide: Test-Time Prior Adaptation for Simulation-Based Inference](https://mlsanity.com/paper/2510.13763/) (2025)
- [DM4CT: Benchmarking Diffusion Models for Computed Tomography Reconstruction](https://mlsanity.com/paper/2602.18589/) (2026)
- [Sample-efficient evidence estimation of score based priors for model selection](https://mlsanity.com/paper/2602.20549/) (2026)
- [Bayesian Conditioned Diffusion Models for Inverse Problems](https://mlsanity.com/paper/2406.09768/) (2024)
- [Enhancing Diffusion Posterior Sampling for Inverse Problems by Integrating Crafted Measurements](https://mlsanity.com/paper/2411.09850/) (2024)
- [Latent Refinement via Flow Matching for Training-free Linear Inverse Problem Solving](https://mlsanity.com/paper/2511.06138/) (2025)
- [Taming Score-Based Denoisers in ADMM: A Convergent Plug-and-Play Framework](https://mlsanity.com/paper/2603.10281/) (2026)
- [Image Restoration via Diffusion Models with Dynamic Resolution](https://mlsanity.com/paper/2605.14267/) (2026)
- [MAP-based Problem-Agnostic diffusion model for Inverse Problems](https://mlsanity.com/paper/2501.15128/) (2025)
- [Noise Conditional Variational Score Distillation](https://mlsanity.com/paper/2506.09416/) (2025)
- [Diffusion models for multivariate subsurface generation and efficient probabilistic inversion](https://mlsanity.com/paper/2507.15809/) (2025)
- [Inverse problems with diffusion models: MAP estimation via mode-seeking loss](https://mlsanity.com/paper/2512.10524/) (2025)
- [Softly Constrained Denoisers for Diffusion Models Applied to Partial Differential Equations](https://mlsanity.com/paper/2512.14980/) (2025)
- [Covariance-aware sampling for Diffusion Models](https://mlsanity.com/paper/2605.13910/) (2026)

## Abstract (arXiv)

Recent diffusion models provide a promising zero-shot solution to noisy linear inverse problems without retraining for specific inverse problems. In this paper, we reveal that recent methods can be uniformly interpreted as employing a Gaussian approximation with hand-crafted isotropic covariance for the intractable denoising posterior to approximate the conditional posterior mean. Inspired by this finding, we propose to improve recent methods by using more principled covariance determined by maximum likelihood estimation. To achieve posterior covariance optimization without retraining, we provide general plug-and-play solutions based on two approaches specifically designed for leveraging pre-trained models with and without reverse covariance. We further propose a scalable method for learning posterior covariance prediction based on representation with orthonormal basis. Experimental results demonstrate that the proposed methods significantly enhance reconstruction performance without requiring hyperparameter tuning.