mlsanityData to 7 Oct 2026

Adam: A Method for Stochastic Optimization

arXiv 1412.6980 ↗First posted 22 Dec 2014cs.LGcited by 36,960

Not on the map. Why?

Cited by 1,084 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.

Key facts. Cited by 36,960 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Dec 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 4; 2018: 24; 2019: 36; 2020: 58; 2021: 8944; 2022: 8705; 2023: 7743; 2024: 5153; 2025: 4004; 2026: 2289.

36,960 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  12. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
38 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  13. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  14. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  15. 2021Vision Transformers for Dense Predictioncited by 1,789
  16. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  17. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  18. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  19. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  20. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  21. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  22. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  23. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  24. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  25. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  26. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  27. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  28. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  29. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  30. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  31. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  32. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  33. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  34. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  35. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  36. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  37. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  38. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131

50 most cited of 36,960 All 36,960 (JSON)