mlsanityData to 14 Oct 2026

Randomized Momentum Transport with Cheap Predictors

arXiv 2610.13789 ↗First posted 9 Oct 2026cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Momentum methods such as STORM and second-order momentum track the gradient by adding a transport correction to momentum, at the cost of an extra oracle call per iteration. We introduce randomized momentum transport, which evaluates the correction with probability q and otherwise uses a cheap prediction, with importance weighting that preserves the conditional mean and adds (q-1-1) times the prediction loss to the variance. Under transport-bias, second-moment, and residual-control assumptions, linear minimization oracle (LMO) updates, including normalized, sign, and exact spectral (Muon) directions, retain an O(T-1/3) stationarity bound for each fixed q>0 with a single-gradient initialization. For unnormalized STORM, some q<1 lowers the leading coefficient of our sufficient gradient budget if and only if the gradient-difference noise modulus exceeds the smoothness constant, by at most about 10%. An online diagonal predictor admits a regret-to-convergence guarantee. At approximately equal oracle cost, querying a quarter of the corrections lowers the final training loss by 18 to 43% on MNIST1D and raises the test accuracy of Hessian-vector transport on CIFAR-10 by 1.5 points.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 24 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

20 of 24 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2018Group Normalizationcited by 1,524
  4. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  5. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  6. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  7. 2020Momentum Improves Normalized SGDcited by 139
  8. 2020PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimizationcited by 119
  9. 2019Distributed Learning with Compressed Gradient Differencescited by 112
  10. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 99
  11. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  12. 2021MARINA: Faster Non-Convex Distributed Learning with Compressioncited by 78
8 more
  1. 2014A Stochastic Quasi-Newton Method for Large-Scale Optimizationcited by 69
  2. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
  3. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 27
  4. 2020Scaling Down Deep Learning with MNIST-1Dcited by 25
  5. 2021Better SGD using Second-order Momentumcited by 15
  6. 2022Second-order optimization with lazy Hessianscited by 15
  7. 2022Momentum-Based Policy Gradient with Second-Order Informationcited by 13
  8. 2025Better LMO-based Momentum Methods with Second-Order Informationcited by 4

Built on by

Not cited yet.