mlsanityData to 14 Oct 2026

RanSOM: Second-Order Momentum with Randomized Scaling for Constrained and Unconstrained Optimization

arXiv 2602.06824 ↗First posted 6 Feb 2026math.OC cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Second-order momentum corrects the lag of momentum behind the current gradient with a Hessian-vector product (HVP), but existing corrections need either a Lipschitz Hessian or extra forward and backward passes at an auxiliary point. We show that randomizing the step length removes this trade-off: if the step is exponentially distributed with mean η, a Stein-type identity makes η times one HVP at the new iterate an estimate of the gradient change that is unbiased in expectation over the step, and this HVP shares its forward pass with the next gradient. The resulting method, RanSOM-E, works with normalized, sign, and Muon-type steps, and a Beta-distributed variant, RanSOM-B, keeps Frank-Wolfe iterates feasible. For objectives with Lipschitz gradient and heavy-tailed gradient and Hessian noise with tail indices p, q ∈ (1,2], RanSOM-E attains the optimal oracle-complexity exponents for every pair (p,q), including p neq q, without a Lipschitz Hessian or clipping (O(varepsilon-3) for p = q = 2). On CIFAR-10 with ResNet-20, we detect no accuracy difference from existing second-order momentum methods, and its mean update time is lower than that of the auxiliary-point method.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

20 of 28 references.

References · 20

  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  2. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  3. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  4. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
  5. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  6. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 287
  7. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  8. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  9. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  10. 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 49
  11. 2024Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clippingcited by 42
  12. 2023Generalized-Smooth Nonconvex Optimization is As Efficient As Smooth Nonconvex Optimizationcited by 35
8 more
  1. 2019One Sample Stochastic Frank-Wolfecited by 32
  2. 2020Scaling Down Deep Learning with MNIST-1Dcited by 25
  3. 2024Random Scaling and Momentum for Non-smooth Non-convex Optimizationcited by 19
  4. 2021Better SGD using Second-order Momentumcited by 15
  5. 2022Momentum-Based Policy Gradient with Second-Order Informationcited by 13
  6. 2024Improving Stochastic Cubic Newton with Momentumcited by 6
  7. 2025Better LMO-based Momentum Methods with Second-Order Informationcited by 4
  8. 2025Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limitscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Stochastic Compositional Optimization via Hybrid Momentum Frank--Wolfecited by 1