# Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence

arXiv: https://arxiv.org/abs/2608.22129 · HTML: https://mlsanity.com/paper/2608.22129/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 75 of its 104 references that are in the map.
- First posted on arXiv 22 Aug 2026.

## Builds on (most cited first, 30 of 75)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [WIRE: Wavelet Implicit Neural Representations](https://mlsanity.com/paper/2301.05187/) (2023)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)
- [SOAP: Improving and Stabilizing Shampoo using Adam](https://mlsanity.com/paper/2409.11321/) (2024)
- [Lower Bounds for Finding Stationary Points I](https://mlsanity.com/paper/1710.11606/) (2017)
- [Why Transformers Need Adam: A Hessian Perspective](https://mlsanity.com/paper/2402.16788/) (2024)
- [Adaptive Gradient Methods at the Edge of Stability](https://mlsanity.com/paper/2207.14484/) (2022)
- [FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions](https://mlsanity.com/paper/2312.02434/) (2023)
- [Sub-sampled Cubic Regularization for Non-convex Optimization](https://mlsanity.com/paper/1705.05933/) (2017)
- [Stochastic Cubic Regularization for Fast Nonconvex Optimization](https://mlsanity.com/paper/1711.02838/) (2017)
- [Acceleration Methods](https://mlsanity.com/paper/2101.09545/) (2021)
- [Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information](https://mlsanity.com/paper/1708.07164/) (2017)
- [How Does Adaptive Optimization Impact Local Neural Network Geometry?](https://mlsanity.com/paper/2211.02254/) (2022)
- [Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivity](https://mlsanity.com/paper/2410.08198/) (2024)
- [Regularized Newton Method with Global O(1/k2) Convergence](https://mlsanity.com/paper/2112.02089/) (2021)
- [Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function](https://mlsanity.com/paper/2205.12856/) (2022)
- [Towards Quantifying the Preconditioning Effect of Adam](https://mlsanity.com/paper/2402.07114/) (2024)
- [Gradient Descent Finds the Cubic-Regularized Non-Convex Newton Step](https://mlsanity.com/paper/1612.00547/) (2016)
- [The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton](https://mlsanity.com/paper/2510.09378/) (2025)
- [Second-order optimization with lazy Hessians](https://mlsanity.com/paper/2212.00781/) (2022)
- [Randomized Block Cubic Newton Method](https://mlsanity.com/paper/1802.04084/) (2018)
- [FreSh: Frequency Shifting for Accelerated Neural Representation Learning](https://mlsanity.com/paper/2410.05050/) (2024)

## Built on by (most cited first, 1 of 1)

- [Physics-Informed Neural Network Surrogate for Oxygen Vacancy Dynamics in epitaxial SrTiO3 on Si memristors via Dynamic Spectral Optimization](https://mlsanity.com/paper/2609.02966/) (2026)

## Abstract (arXiv)

Overparameterized networks often have ill-conditioned, saddle rich loss landscapes. They are usually trained with first-order methods that struggle on those landscapes, while many second-order methods stay infeasible at scale. Some scalable cubic methods replace the Hessian by a computationally cheap approximation and drop the exact block curvature. This paper introduces ARC-block, a blockwise adaptive regularization with cubics (ARC) optimizer. For each parameter tensor it minimizes a cubic model over the true block Hessian, with a per-block adaptive cubic constant and a monotone guard on the full loss. Large tensors are handled matrix-free in a Lanczos-built Krylov subspace. ARC-block's CubicKrylov subsolver's descent is monotone with a per-block cubic decrease bound. For exact steps, and for Krylov steps that meet the Cartis--Gould--Toint residual test, we provide an O(√Bvarepsilon-3/2) sweep bound above a cross-block coupling floor. Another subsolver, ARC-varphi1, uses a trust-controlled exponential relaxation step. Among the cubic Newton methods evaluated here, these are the only ones whose steps stay exact on every parameter block (within the Krylov subspace), including on a 91.4M-parameter implicit neural representation (INR). Run to convergence on FINER 2D image-fitting INR network, ARC-varphi1 reaches 133.5 dB peak signal-to-noise ratio (PSNR), with no post peak collapse. Tuned Adam peaks at 78.2 dB at 2,629 s (step ~625k, about 44 minutes), further steps degrade its PSNR. On the same benchmark, SOAP, L-BFGS, and the scalable diagonal cubic method AdaCubic stay at or below Adam's 78.2 dB peak. Over the benchmark's 20 images, run to a plateau rule, ARC-varphi1's per-image PSNR has median 132.4 and mean 131.5±3.3 dB against tuned Adam's 72.8±4.5 dB.