# Online Deep Equilibrium Learning for Regularization by Denoising

arXiv: https://arxiv.org/abs/2205.13051 · HTML: https://mlsanity.com/paper/2205.13051/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 115 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 30 of 64)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Plug-and-Play Image Restoration with Deep Denoiser Prior](https://mlsanity.com/paper/2008.13751/) (2020)
- [Deep Convolutional Neural Network for Inverse Problems in Imaging](https://mlsanity.com/paper/1611.03679/) (2016)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Learning Deep CNN Denoiser Prior for Image Restoration](https://mlsanity.com/paper/1704.03264/) (2017)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [MoDL: Model Based Deep Learning Architecture for Inverse Problems](https://mlsanity.com/paper/1712.02862/) (2017)
- [The Little Engine that Could: Regularization by Denoising (RED)](https://mlsanity.com/paper/1611.02862/) (2016)
- [Deep Learning Techniques for Inverse Problems in Imaging](https://mlsanity.com/paper/2005.06001/) (2020)
- [Learned Primal-dual Reconstruction](https://mlsanity.com/paper/1707.06474/) (2017)
- [Plug-and-Play ADMM for Image Restoration: Fixed Point Convergence and Applications](https://mlsanity.com/paper/1605.01710/) (2016)
- [Low-Dose CT with a Residual Encoder-Decoder Convolutional Neural Network (RED-CNN)](https://mlsanity.com/paper/1702.00288/) (2017)
- [Plug-and-Play Methods Provably Converge with Properly Trained Denoisers](https://mlsanity.com/paper/1905.05406/) (2019)
- [Non-convex Optimization for Machine Learning](https://mlsanity.com/paper/1712.07897/) (2017)
- [Deep Equilibrium Architectures for Inverse Problems in Imaging](https://mlsanity.com/paper/2102.07944/) (2021)
- [Deep Plug-and-Play Super-Resolution for Arbitrary Blur Kernels](https://mlsanity.com/paper/1903.12529/) (2019)
- [Gradient Step Denoiser for convergent Plug-and-Play](https://mlsanity.com/paper/2110.03220/) (2021)
- [Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems](https://mlsanity.com/paper/1704.03488/) (2017)
- [Regularization by Denoising: Clarifications and New Interpretations](https://mlsanity.com/paper/1806.02296/) (2018)
- [Monotone operator equilibrium networks](https://mlsanity.com/paper/2006.08591/) (2020)
- [Framing U-Net via Deep Convolutional Framelets: Application to Sparse-view CT](https://mlsanity.com/paper/1708.08333/) (2017)

## Built on by (most cited first, 22 of 22)

- [DOLCE: A Model-Based Probabilistic Diffusion Framework for Limited-Angle CT Reconstruction](https://mlsanity.com/paper/2211.12340/) (2022)
- [What's in a Prior? Learned Proximal Networks for Inverse Problems](https://mlsanity.com/paper/2310.14344/) (2023)
- [TorchDEQ: A Library for Deep Equilibrium Models](https://mlsanity.com/paper/2310.18605/) (2023)
- [Learning Image Priors through Patch-based Diffusion Models for Solving Inverse Problems](https://mlsanity.com/paper/2406.02462/) (2024)
- [DiffusionBlend: Learning 3D Image Prior through Position-aware Diffusion Score Blending for 3D Computed Tomography Reconstruction](https://mlsanity.com/paper/2406.10211/) (2024)
- [Differentiating Through Integer Linear Programs with Quadratic Regularization and Davis-Yin Splitting](https://mlsanity.com/paper/2301.13395/) (2023)
- [Prior Mismatch and Adaptation in PnP-ADMM with a Nonconvex Convergence Analysis](https://mlsanity.com/paper/2310.00133/) (2023)
- [PtychoDV: Vision Transformer-Based Deep Unrolling Network for Ptychographic Image Reconstruction](https://mlsanity.com/paper/2310.07504/) (2023)
- [Block Coordinate Plug-and-Play Methods for Blind Inverse Problems](https://mlsanity.com/paper/2305.12672/) (2023)
- [Self-Supervised Scalable Deep Compressed Sensing](https://mlsanity.com/paper/2308.13777/) (2023)
- [Measurement Score-Based Diffusion Model](https://mlsanity.com/paper/2505.11853/) (2025)
- [On Logical Extrapolation for Mazes with Recurrent and Implicit Networks](https://mlsanity.com/paper/2410.03020/) (2024)
- [Robustness of Deep Equilibrium Architectures to Changes in the Measurement Model](https://mlsanity.com/paper/2211.00531/) (2022)
- [Overcoming Distribution Shifts in Plug-and-Play Methods with Test-Time Training](https://mlsanity.com/paper/2403.10374/) (2024)
- [Swap-Net: A Memory-Efficient 2.5D Network for Sparse-View 3D Cone Beam CT Reconstruction](https://mlsanity.com/paper/2410.10836/) (2024)
- [End-to-End Training of High-Dimensional Optimal Control with Implicit Hamiltonians via Jacobian-Free Backpropagation](https://mlsanity.com/paper/2510.00359/) (2025)
- [Efficient Model-Based Deep Learning via Network Pruning and Fine-Tuning](https://mlsanity.com/paper/2311.02003/) (2023)
- [Recovering Pulse Waves from Video Using Deep Unrolling and Deep Equilibrium Models](https://mlsanity.com/paper/2503.17269/) (2025)
- [Self-Supervised Deep Equilibrium Models for Inverse Problems with Theoretical Guarantees](https://mlsanity.com/paper/2210.03837/) (2022)
- [Efficient Unrolled Networks for Large-Scale 3D Inverse Problems](https://mlsanity.com/paper/2601.02141/) (2026)
- [On the Convergence of Jacobian-Free Backpropagation for Optimal Control Problems with Implicit Hamiltonians](https://mlsanity.com/paper/2602.00921/) (2026)
- [Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approach](https://mlsanity.com/paper/2602.03297/) (2026)

## Abstract (arXiv)

Plug-and-Play Priors (PnP) and Regularization by Denoising (RED) are widely-used frameworks for solving imaging inverse problems by computing fixed-points of operators combining physical measurement models and learned image priors. While traditional PnP/RED formulations have focused on priors specified using image denoisers, there is a growing interest in learning PnP/RED priors that are end-to-end optimal. The recent Deep Equilibrium Models (DEQ) framework has enabled memory-efficient end-to-end learning of PnP/RED priors by implicitly differentiating through the fixed-point equations without storing intermediate activation values. However, the dependence of the computational/memory complexity of the measurement models in PnP/RED on the total number of measurements leaves DEQ impractical for many imaging applications. We propose ODER as a new strategy for improving the efficiency of DEQ through stochastic approximations of the measurement models. We theoretically analyze ODER giving insights into its convergence and ability to approximate the traditional DEQ approach. Our numerical results suggest the potential improvements in training/testing complexity due to ODER on three distinct imaging applications.