# Self-concordant smoothing in proximal quasi-Newton algorithms for large-scale convex composite optimization

arXiv: https://arxiv.org/abs/2309.01781 · HTML: https://mlsanity.com/paper/2309.01781/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 76 references that are in the map.
- First posted on arXiv 4 Sep 2023.

## Builds on (most cited first, 17 of 17)

- [Gap Safe screening rules for sparsity enforcing penalties](https://mlsanity.com/paper/1611.05780/) (2016)
- [Generalized Self-Concordant Functions: A Recipe for Newton-Type Methods](https://mlsanity.com/paper/1703.04599/) (2017)
- [Finite-sample analysis of M-estimators using self-concordance](https://mlsanity.com/paper/1810.06838/) (2018)
- [Regularized Newton Method with Global O(1/k2) Convergence](https://mlsanity.com/paper/2112.02089/) (2021)
- [Greedy Quasi-Newton Methods with Explicit Superlinear Convergence](https://mlsanity.com/paper/2002.00657/) (2020)
- [A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems](https://mlsanity.com/paper/1607.05428/) (2016)
- [An efficient Hessian based algorithm for solving large-scale sparse group Lasso problems](https://mlsanity.com/paper/1712.05910/) (2017)
- [GAP Safe Screening Rules for Sparse-Group-Lasso](https://mlsanity.com/paper/1602.06225/) (2016)
- [Forward-backward quasi-Newton methods for nonsmooth optimization problems](https://mlsanity.com/paper/1604.08096/) (2016)
- [Composite Self-Concordant Minimization](https://mlsanity.com/paper/1308.2867/) (2013)
- [Forward-backward envelope for the sum of two nonconvex functions: Further properties and nonmonotone line-search algorithms](https://mlsanity.com/paper/1606.06256/) (2016)
- [On Quasi-Newton Forward--Backward Splitting: Proximal Calculus and Convergence](https://mlsanity.com/paper/1801.08691/) (2018)
- [Two-Layer Feature Reduction for Sparse-Group Lasso via Decomposition of Convex Sets](https://mlsanity.com/paper/1410.4210/) (2014)
- [A Simple and Efficient Algorithm for Nonlinear Model Predictive Control](https://mlsanity.com/paper/1709.06487/) (2017)
- [Sparse Signal Estimation by Maximally Sparse Convex Optimization](https://mlsanity.com/paper/1302.5729/) (2013)
- [Forward-backward truncated Newton methods for convex composite optimization](https://mlsanity.com/paper/1402.6655/) (2014)
- [SCORE: Approximating Curvature Information under Self-Concordant Regularization](https://mlsanity.com/paper/2112.07344/) (2021)

## Built on by (most cited first, 1 of 1)

- [Regularized Gauss-Newton for Optimizing Overparameterized Neural Networks](https://mlsanity.com/paper/2404.14875/) (2024)

## Abstract (arXiv)

We introduce a notion of self-concordant smoothing for minimizing the sum of two convex functions, one of which is smooth and the other nonsmooth. The key highlight is a natural property of the resulting problem's structure that yields a variable metric selection method and a step length rule especially suited to proximal quasi-Newton algorithms. Also, we efficiently handle specific structures promoted by the nonsmooth term, such as l1-regularization and group lasso penalties. A convergence analysis for the class of proximal quasi-Newton methods covered by our framework is presented. In particular, we obtain guarantees, under standard assumptions, for two algorithms: Prox-N-SCORE (a proximal Newton method) and Prox-GGN-SCORE (a proximal generalized Gauss-Newton method). The latter uses a low-rank approximation of the Hessian inverse, reducing most of the cost of matrix inversion and making it effective for overparameterized machine learning models. Numerical experiments on synthetic and real data demonstrate the efficiency of both algorithms against state-of-the-art approaches. A Julia implementation is publicly available at https://github.com/adeyemiadeoye/SelfConcordantSmoothOptimization.jl.