# Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate

arXiv: https://arxiv.org/abs/2401.03058 · HTML: https://mlsanity.com/paper/2401.03058/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 30 references that are in the map.
- First posted on arXiv 5 Jan 2024.

## Builds on (most cited first, 8 of 8)

- [Coordinate Descent Algorithms](https://mlsanity.com/paper/1502.04759/) (2015)
- [RSN: Randomized Subspace Newton](https://mlsanity.com/paper/1905.10874/) (2019)
- [Stochastic Subspace Cubic Newton Method](https://mlsanity.com/paper/2002.09526/) (2020)
- [Randomized Block Cubic Newton Method](https://mlsanity.com/paper/1802.04084/) (2018)
- [Super-Acceleration with Cyclical Step-sizes](https://mlsanity.com/paper/2106.09687/) (2021)
- [First-Order Methods for Nonconvex Quadratic Minimization](https://mlsanity.com/paper/2003.04546/) (2020)
- [Analysis of Krylov Subspace Solutions of Regularized Nonconvex Quadratic Problems](https://mlsanity.com/paper/1806.09222/) (2018)
- [Sketch-and-Project Meets Newton Method: Global mathcal O(k-2) Convergence with Low-Rank Updates](https://mlsanity.com/paper/2305.13082/) (2023)

## Built on by (most cited first, 3 of 3)

- [LancBiO: dynamic Lanczos-aided bilevel optimization via Krylov subspace](https://mlsanity.com/paper/2404.03331/) (2024)
- [Cubic regularized subspace Newton for non-convex optimization](https://mlsanity.com/paper/2406.16666/) (2024)
- [Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence](https://mlsanity.com/paper/2608.22129/) (2026)

## Abstract (arXiv)

Second-order optimization methods, such as cubic regularized Newton methods, are known for their rapid convergence rates; nevertheless, they become impractical in high-dimensional problems due to their substantial memory requirements and computational costs. One promising approach is to execute second-order updates within a lower-dimensional subspace, giving rise to subspace second-order methods. However, the majority of existing subspace second-order methods randomly select subspaces, consequently resulting in slower convergence rates depending on the problem's dimension d. In this paper, we introduce a novel subspace cubic regularized Newton method that achieves a dimension-independent global convergence rate of Oleft(frac1mk+frac1k2right) for solving convex optimization problems. Here, m represents the subspace dimension, which can be significantly smaller than d. Instead of adopting a random subspace, our primary innovation involves performing the cubic regularized Newton update within the Krylov subspace associated with the Hessian and the gradient of the objective function. This result marks the first instance of a dimension-independent convergence rate for a subspace second-order method. Furthermore, when specific spectral conditions of the Hessian are met, our method recovers the convergence rate of a full-dimensional cubic regularized Newton method. Numerical experiments show our method converges faster than existing random subspace methods, especially for high-dimensional problems.