mlsanityData to 14 Oct 2026

AdamD: Improved bias-correction in Adam

arXiv 2110.10828 ↗First posted 20 Oct 2021cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Here I present a small update to the bias-correction term in the Adam optimizer that has the advantage of making smaller gradient updates in the first several steps of training. With the default bias-correction, Adam may actually make larger than requested gradient updates early in training. By only including the well-justified bias-correction of the second moment gradient estimate, vt, and excluding the bias-correction on the first-order estimate, mt, we attain these more desirable gradient update properties in the first series of steps. The default implementation of Adam may be as sensitive as it is to the hyperparameters β1, β2 partially due to the originally proposed bias correction procedure, and its behavior in early steps.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 2 of its 6 references that are in the map. First posted on arXiv 20 Oct 2021.

Builds on

2 of 6 references.

References · 2

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.