# Weight Prediction Boosts the Convergence of AdamW

arXiv: https://arxiv.org/abs/2302.00195 · HTML: https://mlsanity.com/paper/2302.00195/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 1 Feb 2023.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients](https://mlsanity.com/paper/2010.07468/) (2020)
- [Are Transformers More Robust Than CNNs?](https://mlsanity.com/paper/2111.05464/) (2021)
- [Adaptive Gradient Methods with Dynamic Bound of Learning Rate](https://mlsanity.com/paper/1902.09843/) (2019)
- [Scaled ReLU Matters for Training Vision Transformers](https://mlsanity.com/paper/2109.03810/) (2021)
- [Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform](https://mlsanity.com/paper/1809.02839/) (2018)
- [XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training](https://mlsanity.com/paper/1911.04610/) (2019)

## Built on by (most cited first, 8 of 8)

- [XGrad: Boosting Gradient-Based Optimizers With Weight Prediction](https://mlsanity.com/paper/2305.18240/) (2023)
- [Optimizing Large Language Models for ESG Activity Detection in Financial Texts](https://mlsanity.com/paper/2502.21112/) (2025)
- [PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction](https://mlsanity.com/paper/2312.00839/) (2023)
- [One Step Learning, One Step Review](https://mlsanity.com/paper/2401.10962/) (2024)
- [TabNSA: Native Sparse Attention for Efficient Tabular Data Learning](https://mlsanity.com/paper/2503.09850/) (2025)
- [AdaPlus: Integrating Nesterov Momentum and Precise Stepsize Adjustment on AdamW Basis](https://mlsanity.com/paper/2309.01966/) (2023)
- [SpectralTrain: A Universal Framework for Hyperspectral Image Classification](https://mlsanity.com/paper/2511.16084/) (2025)
- [Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning](https://mlsanity.com/paper/2604.22838/) (2026)

## Abstract (arXiv)

In this paper, we introduce weight prediction into the AdamW optimizer to boost its convergence when training the deep neural network (DNN) models. In particular, ahead of each mini-batch training, we predict the future weights according to the update rule of AdamW and then apply the predicted future weights to do both forward pass and backward propagation. In this way, the AdamW optimizer always utilizes the gradients w.r.t. the future weights instead of current weights to update the DNN parameters, making the AdamW optimizer achieve better convergence. Our proposal is simple and straightforward to implement but effective in boosting the convergence of DNN training. We performed extensive experimental evaluations on image classification and language modeling tasks to verify the effectiveness of our proposal. The experimental results validate that our proposal can boost the convergence of AdamW and achieve better accuracy than AdamW when training the DNN models.