# Dynamically Weighted Momentum with Adaptive Step Sizes for Efficient Deep Network Training

arXiv: https://arxiv.org/abs/2510.25042 · HTML: https://mlsanity.com/paper/2510.25042/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 114 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://mlsanity.com/paper/1705.08292/) (2017)
- [Is Local SGD Better than Minibatch SGD?](https://mlsanity.com/paper/2002.07839/) (2020)
- [CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking](https://mlsanity.com/paper/2303.00332/) (2023)
- [Physics-Driven Deep Learning for Computational Magnetic Resonance Imaging](https://mlsanity.com/paper/2203.12215/) (2022)
- [Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100](https://mlsanity.com/paper/2108.04062/) (2021)
- [Abs-CAM: A Gradient Optimization Interpretable Approach for Explanation of Convolutional Neural Networks](https://mlsanity.com/paper/2207.03648/) (2022)
- [Algorithmic Instabilities of Accelerated Gradient Descent](https://mlsanity.com/paper/2102.02167/) (2021)
- [Knowledge Relation Rank Enhanced Heterogeneous Learning Interaction Modeling for Neural Graph Forgetting Knowledge Tracing](https://mlsanity.com/paper/2304.03945/) (2023)
- [Knowledge Graph Enhanced Intelligent Tutoring System Based on Exercise Representativeness and Informativeness](https://mlsanity.com/paper/2307.15076/) (2023)
- [Memory Augmented Optimizers for Deep Learning](https://mlsanity.com/paper/2106.10708/) (2021)
- [Spatio-Temporal Representation Learning Enhanced Source Cell-phone Recognition from Speech Recordings](https://mlsanity.com/paper/2208.12753/) (2022)
- [Leveraging Label Semantics and Meta-Label Refinement for Multi-Label Question Classification](https://mlsanity.com/paper/2411.01841/) (2024)
- [UKTF: Unified Knowledge Tracing Framework for Subjective and Objective Assessments](https://mlsanity.com/paper/2411.05325/) (2024)

## Abstract (arXiv)

Within the current sphere of deep learning research, despite the extensive application of optimization algorithms such as Stochastic Gradient Descent (SGD) and Adaptive Moment Estimation (Adam), there remains a pronounced inadequacy in their capability to address fluctuations in learning efficiency, meet the demands of complex models, and tackle non-convex optimization issues. These challenges primarily arise from the algorithms' limitations in handling complex data structures and models, for instance, difficulties in selecting an appropriate learning rate, avoiding local optima, and navigating through high-dimensional spaces. To address these issues, this paper introduces a novel optimization algorithm named DWMGrad. This algorithm, building on the foundations of traditional methods, incorporates a dynamic guidance mechanism reliant on historical data to dynamically update momentum and learning rates. This allows the optimizer to flexibly adjust its reliance on historical information, adapting to various training scenarios. This strategy not only enables the optimizer to better adapt to changing environments and task complexities but also, as validated through extensive experimentation, demonstrates DWMGrad's ability to achieve faster convergence rates and higher accuracies under a multitude of scenarios.