# A General Framework for Fast Stagewise Algorithms

arXiv: https://arxiv.org/abs/1408.5801 · HTML: https://mlsanity.com/paper/1408.5801/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Aug 2014.

## Built on by (most cited first, 11 of 11)

- [ResMem: Learn what you can and memorize the rest](https://mlsanity.com/paper/2302.01576/) (2023)
- [Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes](https://mlsanity.com/paper/2406.06838/) (2024)
- [Sparse Bayesian Learning via Stepwise Regression](https://mlsanity.com/paper/2106.06095/) (2021)
- [SGL: Spectral Graph Learning from Measurements](https://mlsanity.com/paper/2104.07867/) (2021)
- [Fast Robust Kernel Regression through Sign Gradient Descent with Early Stopping](https://mlsanity.com/paper/2306.16838/) (2023)
- [SF-SGL: Solver-Free Spectral Graph Learning from Linear Measurements](https://mlsanity.com/paper/2302.04384/) (2023)
- [Tree-Guided Rare Feature Selection and Logic Aggregation with Electronic Health Records Data](https://mlsanity.com/paper/2206.09107/) (2022)
- [Performance of ℓ1 Regularization for Sparse Convex Optimization](https://mlsanity.com/paper/2307.07405/) (2023)
- [Stagewise Boosting Distributional Regression](https://mlsanity.com/paper/2405.18288/) (2024)
- [New explanations and inference for least angle regression](https://mlsanity.com/paper/2602.02491/) (2026)
- [Virtual Dummies: Enabling Scalable FDR-Controlled Variable Selection via Sequential Sampling of Null Features](https://mlsanity.com/paper/2604.07464/) (2026)

## Abstract (arXiv)

Forward stagewise regression follows a very simple strategy for constructing a sequence of sparse regression estimates: it starts with all coefficients equal to zero, and iteratively updates the coefficient (by a small amount ε) of the variable that achieves the maximal absolute inner product with the current residual. This procedure has an interesting connection to the lasso: under some conditions, it is known that the sequence of forward stagewise estimates exactly coincides with the lasso path, as the step size ε goes to zero. Furthermore, essentially the same equivalence holds outside of least squares regression, with the minimization of a differentiable convex loss function subject to an ℓ1 norm constraint (the stagewise algorithm now updates the coefficient corresponding to the maximal absolute component of the gradient). Even when they do not match their ℓ1-constrained analogues, stagewise estimates provide a useful approximation, and are computationally appealing. Their success in sparse modeling motivates the question: can a simple, effective strategy like forward stagewise be applied more broadly in other regularization settings, beyond the ℓ1 norm and sparsity? The current paper is an attempt to do just this. We present a general framework for stagewise estimation, which yields fast algorithms for problems such as group-structured learning, matrix completion, image denoising, and more.