mlsanityData to 14 Oct 2026

Torque-Aware Momentum

arXiv 2412.18790 ↗First posted 25 Dec 2024cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Efficiently exploring complex loss landscapes is key to the performance of deep neural networks. While momentum-based optimizers are widely used in state-of-the-art setups, classical momentum can still struggle with large, misaligned gradients, leading to oscillations. To address this, we propose Torque-Aware Momentum (TAM), which introduces a damping factor based on the angle between the new gradients and previous momentum, stabilizing the update direction during training. Empirical results show that TAM, which can be combined with both SGD and Adam, enhances exploration, handles distribution shifts more effectively, and improves generalization performance across various tasks, including image classification and large language model fine-tuning, when compared to classical momentum-based optimizers.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 25 Dec 2024.

Builds on

31 of 48 references.

References · 31

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  3. 2016Learning without Forgettingcited by 2,496
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  6. 2018Link Prediction Based on Graph Neural Networkscited by 832
  7. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  8. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  9. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  10. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  11. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  12. 2020Momentum Improves Normalized SGDcited by 139
19 more
  1. 2020An Improved Analysis of Stochastic Gradient Descent with Momentumcited by 119
  2. 2022The CLEAR Benchmark: Continual LEArning on Real-World Imagerycited by 93
  3. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 86
  4. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 80
  5. 2022When Do Flat Minima Optimizers Work?cited by 78
  6. 2017Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descentcited by 66
  7. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 65
  8. 2024Normalization and effective learning rates in reinforcement learningcited by 58
  9. 2018A Unified Analysis of Stochastic Momentum Methods for Deep Learningcited by 55
  10. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 50
  11. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 40
  12. 2018Aggregated Momentum: Stability Through Passive Dampingcited by 36
  13. 2024Learning Continually by Spectral Regularizationcited by 34
  14. 2021Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalizationcited by 28
  15. 2020LaProp: Separating Momentum and Adaptivity in Adamcited by 20
  16. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 19
  17. 2021Better SGD using Second-order Momentumcited by 15
  18. 2023Do You Remember? Overcoming Catastrophic Forgetting for Fake Audio Detectioncited by 11
  19. 2021AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networkscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizerscited by 0
  2. 2026AURA: Angular Update Rate Adaptation for training complex-valued neural networkscited by 0