# A multiobjective continuation method to compute the regularization path of deep neural networks

arXiv: https://arxiv.org/abs/2308.12044 · HTML: https://mlsanity.com/paper/2308.12044/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 49 references that are in the map.
- First posted on arXiv 23 Aug 2023.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Scalable Pareto Front Approximation for Deep Multi-Objective Learning](https://mlsanity.com/paper/2103.13392/) (2021)
- [Orthant Based Proximal Stochastic Gradient Method for ℓ1-Regularized Optimization](https://mlsanity.com/paper/2004.03639/) (2020)
- [A Bregman Learning Framework for Sparse Neural Networks](https://mlsanity.com/paper/2105.04319/) (2021)
- [Gradient-Based Multiobjective Optimization with Uncertainties](https://mlsanity.com/paper/1612.03815/) (2016)
- [Exploring Structural Sparsity of Deep Networks via Inverse Scale Spaces](https://mlsanity.com/paper/1905.09449/) (2019)
- [On the Treatment of Optimization Problems with L1 Penalty Terms via Multiobjective Continuation](https://mlsanity.com/paper/2012.07483/) (2020)
- [Momentum-based Gradient Methods in Multi-Objective Recommendation](https://mlsanity.com/paper/2009.04695/) (2020)
- [An accelerated proximal gradient method for multiobjective optimization](https://mlsanity.com/paper/2202.10994/) (2022)
- [Conformalization of Sparse Generalized Linear Models](https://mlsanity.com/paper/2307.05109/) (2023)
- [DessiLBI: Exploring Structural Sparsity of Deep Networks via Differential Inclusion Paths](https://mlsanity.com/paper/2007.02010/) (2020)
- [Fast Multiobjective Gradient Methods with Nesterov Acceleration via Inertial Gradient-like Systems](https://mlsanity.com/paper/2207.12707/) (2022)

## Built on by (most cited first, 3 of 3)

- [Multi-objective Deep Learning: Taxonomy and Survey of the State of the Art](https://mlsanity.com/paper/2412.01566/) (2024)
- [Common pitfalls to avoid while using multiobjective optimization in machine learning](https://mlsanity.com/paper/2405.01480/) (2024)
- [Interactive Pareto navigation for deep multi-task learning](https://mlsanity.com/paper/2606.19521/) (2026)

## Abstract (arXiv)

Sparsity is a highly desired feature in deep neural networks (DNNs) since it ensures numerical efficiency, improves the interpretability of models (due to the smaller number of relevant features), and robustness. For linear models, it is well known that there exists a regularization path connecting the sparsest solution in terms of the ℓ1 norm, i.e., zero weights and the non-regularized solution. Very recently, there was a first attempt to extend the concept of regularization paths to DNNs by means of treating the empirical loss and sparsity (ℓ1 norm) as two conflicting criteria and solving the resulting multiobjective optimization problem for low-dimensional DNN. However, due to the non-smoothness of the ℓ1 norm and the high number of parameters, this approach is not very efficient from a computational perspective for high-dimensional DNNs. To overcome this limitation, we present an algorithm that allows for the approximation of the entire Pareto front for the above-mentioned objectives in a very efficient manner for high-dimensional DNNs with millions of parameters. We present numerical examples using both deterministic and stochastic gradients. We furthermore demonstrate that knowledge of the regularization path allows for a well-generalizing network parametrization. To the best of our knowledge, this is the first algorithm to compute the regularization path for non-convex multiobjective optimization problems (MOPs) with millions of degrees of freedom.