Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence

Overparameterized networks often have ill-conditioned, saddle rich loss landscapes. They are usually trained with first-order methods that struggle on those landscapes, while many second-order methods stay infeasible at scale. Some scalable cubic methods replace the Hessian by a computationally cheap approximation and drop the exact block curvature. This paper introduces ARC-block, a blockwise adaptive regularization with cubics (ARC) optimizer. For each parameter tensor it minimizes a cubic model over the true block Hessian, with a per-block adaptive cubic constant and a monotone guard on the full loss. Large tensors are handled matrix-free in a Lanczos-built Krylov subspace. ARC-block's CubicKrylov subsolver's descent is monotone with a per-block cubic decrease bound. For exact steps, and for Krylov steps that meet the Cartis--Gould--Toint residual test, we provide an O(√Bvarepsilon-3/2) sweep bound above a cross-block coupling floor. Another subsolver, ARC-varphi1, uses a trust-controlled exponential relaxation step. Among the cubic Newton methods evaluated here, these are the only ones whose steps stay exact on every parameter block (within the Krylov subspace), including on a 91.4M-parameter implicit neural representation (INR). Run to convergence on FINER 2D image-fitting INR network, ARC-varphi1 reaches 133.5 dB peak signal-to-noise ratio (PSNR), with no post peak collapse. Tuned Adam peaks at 78.2 dB at 2,629 s (step ~625k, about 44 minutes), further steps degrade its PSNR. On the same benchmark, SOAP, L-BFGS, and the scalable diagonal cubic method AdaCubic stay at or below Adam's 78.2 dB peak. Over the benchmark's 20 images, run to a plateau rule, ARC-varphi1's per-image PSNR has median 132.4 and mean 131.5±3.3 dB against tuned Adam's 72.8±4.5 dB.