mlsanityData to 14 Oct 2026

Amos: An Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scale

arXiv 2210.11693 ↗First posted 21 Oct 2022cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We present Amos, a stochastic gradient-based optimizer designed for training deep neural networks. It can be viewed as an Adam optimizer with theoretically supported, adaptive learning-rate decay and weight decay. A key insight behind Amos is that it leverages model-specific information to determine the initial learning-rate and decaying schedules. When used for pre-training BERT variants and T5, Amos consistently converges faster than the state-of-the-art settings of AdamW, achieving better validation loss within <=70% training steps and time, while requiring <=51% memory for slot variables. Our code is open-sourced at: https://github.com/google-research/jestimator

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 25 of its 41 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

25 of 41 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2016Layer Normalizationcited by 5,669
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
13 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  2. 2018Self-Attention with Relative Position Representationscited by 1,120
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  5. 2019On the Convergence of Adam and Beyondcited by 841
  6. 2019Introduction to Online Convex Optimizationcited by 803
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  8. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  9. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  10. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  11. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  12. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  13. 2021Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 1; 2025: 1; 2026: 2.

7 papers.

  1. 2023The Falcon Series of Open Language Modelscited by 535
  2. 2025Accelerating Neural Network Training: An Analysis of the AlgoPerf Competitioncited by 32
  3. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 15
  4. 2023Improving Deep Learning Optimization through Constrained Parameter Regularizationcited by 5
  5. 2026Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluationscited by 3
  6. 2023A Theoretical and Empirical Study on the Convergence of Adam with an "Exact" Constant Step Size in Non-Convex Settingscited by 0
  7. 2024Poisson Process for Bayesian Optimizationcited by 0