Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architectures
Not on the map. Why?
Abstract (arXiv)
We present a general variational framework for the training of freeform nonlinearities in layered computational architectures subject to some slope constraints. The regularization that we add to the traditional training loss penalizes the second-order total variation of each trainable activation. The slope constraints allow us to impose properties such as 1-Lipschitz stability, firm non-expansiveness, and monotonicity/invertibility. These properties are crucial to ensure the proper functioning of certain classes of signal-processing algorithms (e.g., plug-and-play schemes, unrolled proximal gradient, invertible flows). We prove that the global optimum of the stated constrained-optimization problem is achieved with nonlinearities that are adaptive nonuniform linear splines. We then show how to solve the resulting function-optimization problem numerically by representing the nonlinearities in a suitable (nonuniform) B-spline basis. Finally, we illustrate the use of our framework with the data-driven design of (weakly) convex regularizers for the denoising of images and the resolution of inverse problems.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 28 of its 69 references that are in the map. First posted on arXiv 23 Aug 2024.
Builds on
28 of 69 references.
References · 28
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2021Convex optimizationcited by 2,515
- 2016Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoisingcited by 1,358
- 2019Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processingcited by 421
- 2017MoDL: Model Based Deep Learning Architecture for Inverse Problemscited by 387
- 2017Learning a Variational Network for Reconstruction of Accelerated MRI Datacited by 342
- 2016Plug-and-Play ADMM for Image Restoration: Fixed Point Convergence and Applicationscited by 246
- 2019Plug-and-Play Methods Provably Converge with Properly Trained Denoiserscited by 198
- 2015Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restorationcited by 195
- 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 155
- 2021Deep Equilibrium Architectures for Inverse Problems in Imagingcited by 127
- 2022Plug-and-Play Methods for Integrating Physical and Learned Models in Computational Imagingcited by 96
16 more
- 2020A survey on modern trainable activation functionscited by 89
- 2014Learning Activation Functions to Improve Deep Neural Networkscited by 63
- 2018Neural Proximal Gradient Descent for Compressive Imagingcited by 46
- 2022A Neural-Network-Based Convex Regularizer for Inverse Problemscited by 41
- 2018A representer theorem for deep neural networkscited by 38
- 2018A characterization of proximity operatorscited by 32
- 2019HINT: Hierarchical Invertible Neural Transport for Density Estimation and Bayesian Inferencecited by 31
- 2023Learning Weakly Convex Regularizers for Convergent Image-Reconstruction Algorithmscited by 31
- 2019Rethinking the CSC Model for Natural Imagescited by 21
- 2014Insights into analysis operator learning: From patch-based sparse models to higher-order MRFscited by 18
- 2020Sparsest Piecewise-Linear Regression of One-Dimensional Datacited by 12
- 2015The Alternating Descent Conditional Gradient Method for Sparse Inverse Problemscited by 11
- 2014Learning ℓ1-based analysis and synthesis sparsity priors using bi-level optimizationcited by 6
- 2017Learning Convex Regularizers for Optimal Bayesian Denoisingcited by 5
- 2018A Greedy Approach to ℓ0,∞ Based Convolutional Sparse Codingcited by 2
- 2021Sparsest Univariate Learning Models Under Lipschitz Constraintcited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.