mlsanityData to 13 Oct 2026

Quasi-hyperbolic momentum and Adam for deep learning

arXiv 1810.06801 ↗First posted 16 Oct 2018cs.LG stat.MLcited by 48

Not on the map. Why?

Abstract (arXiv)

Momentum-based acceleration of stochastic gradient descent (SGD) is widely used in deep learning. We propose the quasi-hyperbolic momentum algorithm (QHM) as an extremely simple alteration of momentum SGD, averaging a plain SGD step with a momentum step. We describe numerous connections to and identities with other algorithms, and we characterize the set of two-state optimization algorithms that QHM can recover. Finally, we propose a QH variant of Adam called QHAdam, and we empirically demonstrate that our algorithms lead to significantly improved training in a variety of settings, including a new state-of-the-art result on WMT16 EN-DE. We hope that these empirical results, combined with the conceptual and practical simplicity of QHM and QHAdam, will spur interest from both practitioners and researchers. Code is immediately available.

Key facts. Cited by 48 papers in the map (data through 13 Oct 2026). First posted on arXiv 16 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 10; 2022: 7; 2023: 7; 2024: 6; 2025: 7; 2026: 11.

48 papers.

Sort:

By area:

  1. 2023Symbolic Discovery of Optimization Algorithmscited by 425
  2. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 101
  3. 2021NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learningcited by 88
  4. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  5. 2021DANets: Deep Abstract Networks for Tabular Data Classification and Regressioncited by 50
  6. 2022T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interactioncited by 41
  7. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16
  8. 2023On the Role of Server Momentum in Federated Learningcited by 16
  9. 2025Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulatorcited by 14
  10. 2024Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performancecited by 13
  11. 20220/1 Deep Neural Networks via Block Coordinate Descentcited by 10
  12. 2022On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learningcited by 10
36 more
  1. 2021Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimizationcited by 8
  2. 2023Stochastic Parrots Looking for Stochastic Parrots: LLMs are Easy to Fine-Tune and Hard to Detect with other LLMscited by 8
  3. 2024On the Performance Analysis of Momentum Method: A Frequency Domain Perspectivecited by 8
  4. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 8
  5. 2021Provable Convergence of Nesterov's Accelerated Gradient Method for Over-Parameterized Neural Networkscited by 7
  6. 2022Self-Supervised Visual Representation Learning via Residual Momentumcited by 7
  7. 2021Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimizationcited by 6
  8. 2021Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problemscited by 5
  9. 2022Tuning Stochastic Gradient Algorithms for Statistical Inference via Large-Sample Asymptoticscited by 4
  10. 2023Navigating the Noise: Bringing Clarity to ML Parameterization Design with O(100) Ensemblescited by 3
  11. 2023Enhancing Deep Neural Network Training Efficiency and Performance through Linear Predictioncited by 3
  12. 2021Gradient Temporal Difference with Momentum: Stability and Convergencecited by 2
  13. 2021Variance Reduction in Deep Learning: More Momentum is All You Needcited by 2
  14. 2024Parallel Momentum Methods Under Biased Gradient Estimationscited by 2
  15. 2024l0-Regularized Sparse Coding-based Interpretable Network for Multi-Modal Image Fusioncited by 2
  16. 2025Increasing Batch Size Improves Convergence of Stochastic Gradient Descent with Momentumcited by 2
  17. 2025MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updatescited by 2
  18. 2025Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradientscited by 2
  19. 2026Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methodscited by 2
  20. 2021Scaling transition from momentum stochastic gradient descent to plain stochastic gradient descentcited by 1
  21. 2025Both Asymptotic and Non-Asymptotic Convergence of Quasi-Hyperbolic Momentum using Increasing Batch Sizecited by 1
  22. 2026Dynamic Momentum Recalibration in Online Gradient Learningcited by 1
  23. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
  24. 2022Multilevel-in-Layer Training for Deep Neural Network Regressioncited by 0
  25. 2023Fine-Tuning Adaptive Stochastic Optimizers: Determining the Optimal Hyperparameter ε via Gradient Magnitude Histogram Analysiscited by 0
  26. 2024A Selective Review on Statistical Methods for Massive Data Computation: Distributed Computing, Subsampling, and Minibatch Techniquescited by 0
  27. 2024Neural Optimizer Equation, Decay Function, and Learning Rate Schedule Joint Evolutioncited by 0
  28. 2025Hyperbolic Optimizationcited by 0
  29. 2025SVRG and Beyond via Posterior Correctioncited by 0
  30. 2026Gradient Extrapolation-Based Policy Optimizationcited by 0
  31. 2026OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizerscited by 0
  32. 2026Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Studycited by 0
  33. 2026Enhanced Artificial Neural Networks Using QHAdamW in Air Quality Forecastingcited by 0
  34. 2026A Journey to the Edge of Stabilitycited by 0
  35. 2026Relative Generalization Invariance of LLM Pretrainingcited by 0
  36. 2026Second-Moment Stochastic Approximation Methodscited by 0