# Learning Sparsity-Promoting Regularizers using Bilevel Optimization

arXiv: https://arxiv.org/abs/2207.08939 · HTML: https://mlsanity.com/paper/2207.08939/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 66 references that are in the map.
- First posted on arXiv 18 Jul 2022.

## Builds on (most cited first, 21 of 21)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Deep Convolutional Neural Network for Inverse Problems in Imaging](https://mlsanity.com/paper/1611.03679/) (2016)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [The Little Engine that Could: Regularization by Denoising (RED)](https://mlsanity.com/paper/1611.02862/) (2016)
- [Plug-and-Play ADMM for Image Restoration: Fixed Point Convergence and Applications](https://mlsanity.com/paper/1605.01710/) (2016)
- [Deep Equilibrium Architectures for Inverse Problems in Imaging](https://mlsanity.com/paper/2102.07944/) (2021)
- [Differentiating Through a Cone Program](https://mlsanity.com/paper/1904.09043/) (2019)
- [Image Reconstruction: From Sparsity to Data-adaptive Methods and Machine Learning](https://mlsanity.com/paper/1904.02816/) (2019)
- [Plug-and-Play Unplugged: Optimization Free Reconstruction using Consensus Equilibrium](https://mlsanity.com/paper/1705.08983/) (2017)
- [Recovery Analysis for Plug-and-Play Priors using the Restricted Eigenvalue Condition](https://mlsanity.com/paper/2106.03668/) (2021)
- [On Correctness of Automatic Differentiation for Non-Differentiable Functions](https://mlsanity.com/paper/2006.06903/) (2020)
- [Bilevel methods for image reconstruction](https://mlsanity.com/paper/2109.09610/) (2021)
- [Insights into analysis operator learning: From patch-based sparse models to higher-order MRFs](https://mlsanity.com/paper/1401.2804/) (2014)
- [Bilevel parameter learning for higher-order total variation regularisation models](https://mlsanity.com/paper/1508.07243/) (2015)
- [ADMM-DAD net: a deep unfolding network for analysis compressed sensing](https://mlsanity.com/paper/2110.06986/) (2021)
- [Learning ℓ1-based analysis and synthesis sparsity priors using bi-level optimization](https://mlsanity.com/paper/1401.4105/) (2014)
- [Model-based Reconstruction with Learning: From Unsupervised to Supervised and Beyond](https://mlsanity.com/paper/2103.14528/) (2021)
- [The Representation Power of Neural Networks: Breaking the Curse of Dimensionality](https://mlsanity.com/paper/2012.05451/) (2020)
- [Data-Driven Learning of a Union of Sparsifying Transforms Model for Blind Compressed Sensing](https://mlsanity.com/paper/1511.01289/) (2015)
- [Steepest descent algorithm on orthogonal Stiefel manifolds](https://mlsanity.com/paper/1709.06295/) (2017)
- [Wavelet Domain Residual Network (WavResNet) for Low-Dose X-ray CT Reconstruction](https://mlsanity.com/paper/1703.01383/) (2017)

## Built on by (most cited first, 4 of 4)

- [Learning sparsity-promoting regularizers for linear inverse problems](https://mlsanity.com/paper/2412.16031/) (2024)
- [Pruning Unrolled Networks (PUN) at Initialization for MRI Reconstruction Improves Generalization](https://mlsanity.com/paper/2412.18668/) (2024)
- [Learning Explicitly Conditioned Sparsifying Transforms](https://mlsanity.com/paper/2403.03168/) (2024)
- [Meta-Prior: Meta learning for Adaptive Inverse Problem Solvers](https://mlsanity.com/paper/2311.18710/) (2023)

## Abstract (arXiv)

We present a method for supervised learning of sparsity-promoting regularizers for denoising signals and images. Sparsity-promoting regularization is a key ingredient in solving modern signal reconstruction problems; however, the operators underlying these regularizers are usually either designed by hand or learned from data in an unsupervised way. The recent success of supervised learning (mainly convolutional neural networks) in solving image reconstruction problems suggests that it could be a fruitful approach to designing regularizers. Towards this end, we propose to denoise signals using a variational formulation with a parametric, sparsity-promoting regularizer, where the parameters of the regularizer are learned to minimize the mean squared error of reconstructions on a training set of ground truth image and measurement pairs. Training involves solving a challenging bilievel optimization problem; we derive an expression for the gradient of the training loss using the closed-form solution of the denoising problem and provide an accompanying gradient descent algorithm to minimize it. Our experiments with structured 1D signals and natural images show that the proposed method can learn an operator that outperforms well-known regularizers (total variation, DCT-sparsity, and unsupervised dictionary learning) and collaborative filtering for denoising. While the approach we present is specific to denoising, we believe that it could be adapted to the larger class of inverse problems with linear measurement models, giving it applicability in a wide range of signal reconstruction settings.