# Restarted contractive operators to learn at equilibrium

arXiv: https://arxiv.org/abs/2506.13239 · HTML: https://mlsanity.com/paper/2506.13239/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 16 Jun 2025.

## Builds on (most cited first, 21 of 21)

- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Plug-and-Play Image Restoration with Deep Denoiser Prior](https://mlsanity.com/paper/2008.13751/) (2020)
- [Deep Equilibrium Architectures for Inverse Problems in Imaging](https://mlsanity.com/paper/2102.07944/) (2021)
- [Plug-and-Play Methods for Integrating Physical and Learned Models in Computational Imaging](https://mlsanity.com/paper/2203.17061/) (2022)
- [JFB: Jacobian-Free Backpropagation for Implicit Networks](https://mlsanity.com/paper/2103.12803/) (2021)
- [Proximal Denoiser for Convergent Plug-and-Play Optimization with Nonconvex Regularization](https://mlsanity.com/paper/2201.13256/) (2022)
- [Nonsmooth Implicit Differentiation for Machine Learning and Optimization](https://mlsanity.com/paper/2106.04350/) (2021)
- [Implicit differentiation of Lasso-type models for hyperparameter optimization](https://mlsanity.com/paper/2002.08943/) (2020)
- [Implicit differentiation for fast hyperparameter selection in non-smooth convex learning](https://mlsanity.com/paper/2105.01637/) (2021)
- [Bilevel methods for image reconstruction](https://mlsanity.com/paper/2109.09610/) (2021)
- [Deep Equilibrium Learning of Explicit Regularizers for Imaging Inverse Problems](https://mlsanity.com/paper/2303.05386/) (2023)
- [UNSURE: self-supervised learning with Unknown Noise level and Stein's Unbiased Risk Estimate](https://mlsanity.com/paper/2409.01985/) (2024)
- [A deep primal-dual proximal network for image restoration](https://mlsanity.com/paper/2007.00959/) (2020)
- [Linearly convergent bilevel optimization with single-step inner methods](https://mlsanity.com/paper/2205.04862/) (2022)
- [Deep Unfolding of the DBFB Algorithm with Application to ROI CT Imaging with Limited Angular Density](https://mlsanity.com/paper/2209.13264/) (2022)
- [Exact worst-case convergence rates of the proximal gradient method for composite convex minimization](https://mlsanity.com/paper/1705.04398/) (2017)
- [Automated data-driven selection of the hyperparameters for Total-Variation based texture segmentation](https://mlsanity.com/paper/2004.09434/) (2020)
- [Single-loop methods for bilevel parameter learning in inverse imaging](https://mlsanity.com/paper/2408.08123/) (2024)
- [An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation](https://mlsanity.com/paper/2412.06436/) (2024)
- [Theoretical and numerical comparison of first-order algorithms for cocoercive equations and smooth convex optimization](https://mlsanity.com/paper/2101.06152/) (2021)
- [Implicit Differentiation for Hyperparameter Tuning the Weighted Graphical Lasso](https://mlsanity.com/paper/2307.02130/) (2023)

## Built on by (most cited first, 2 of 2)

- [Equivariant Splitting: Self-supervised learning from incomplete data](https://mlsanity.com/paper/2510.00929/) (2025)
- [Bilevel Learning via Inexact Stochastic Gradient Descent](https://mlsanity.com/paper/2511.06774/) (2025)

## Abstract (arXiv)

Bilevel optimization offers a methodology to learn hyperparameters in imaging inverse problems, yet its integration with automatic differentiation techniques remains challenging. On the one hand, inverse problems are typically solved by iterating arbitrarily many times some elementary scheme which maps any point to the minimizer of an energy functional, known as equilibrium point. On the other hand, introducing parameters to be learned in the energy functional yield architectures very reminiscent of Neural Networks (NN) known as Unrolled NN and thus suggests the use of Automatic Differentiation (AD) techniques. Yet, applying AD requires for the NN to be of relatively small depth, thus making necessary to truncate an unrolled scheme to a finite number of iterations. First, we show that, at the minimizer, the optimal gradient descent step computed in the Deep Equilibrium (DEQ) framework admits an approximation, known as Jacobian Free Backpropagation (JFB), that is much easier to compute and can be made arbitrarily good by controlling Lipschitz properties of the truncated unrolled scheme. Second, we introduce an algorithm that combines a restart strategy with JFB computed by AD and we show that the learned steps can be made arbitrarily close to the optimal DEQ framework. Third, we complement the theoretical analysis by applying the proposed method to a variety of problems in imaging that progressively depart from the theoretical framework. In particular we show that this method is effective for training weights in weighted norms; stepsizes and regularization levels of Plug-and-Play schemes; and a DRUNet denoiser embedded in Forward-Backward iterates.