# Gradient-based Regularization Parameter Selection for Problems with Non-smooth Penalty Functions

arXiv: https://arxiv.org/abs/1703.09813 · HTML: https://mlsanity.com/paper/1703.09813/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2017.

## Built on by (most cited first, 3 of 3)

- [Efficient differentiable quadratic programming layers: an ADMM approach](https://mlsanity.com/paper/2112.07464/) (2021)
- [Moreau Envelope for Nonconvex Bi-Level Optimization: A Single-loop and Hessian-free Solution Strategy](https://mlsanity.com/paper/2405.09927/) (2024)
- [Value Function Based Difference-of-Convex Algorithm for Bilevel Hyperparameter Selection Problems](https://mlsanity.com/paper/2206.05976/) (2022)

## Abstract (arXiv)

In high-dimensional and/or non-parametric regression problems, regularization (or penalization) is used to control model complexity and induce desired structure. Each penalty has a weight parameter that indicates how strongly the structure corresponding to that penalty should be enforced. Typically the parameters are chosen to minimize the error on a separate validation set using a simple grid search or a gradient-free optimization method. It is more efficient to tune parameters if the gradient can be determined, but this is often difficult for problems with non-smooth penalty functions. Here we show that for many penalized regression problems, the validation loss is actually smooth almost-everywhere with respect to the penalty parameters. We can therefore apply a modified gradient descent algorithm to tune parameters. Through simulation studies on example regression problems, we find that increasing the number of penalty parameters and tuning them using our method can decrease the generalization error.