mlsanityData to 7 Oct 2026

Decoupled Weight Decay Regularization

arXiv 1711.05101 ↗First posted 14 Nov 2017cs.LG cs.NE math.OCcited by 1,722

Not on the map. Why?

Cited by 213 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

L2 regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is not the case for adaptive gradient algorithms, such as Adam. While common implementations of these algorithms employ L2 regularization (often calling it "weight decay" in what may be misleading due to the inequivalence we expose), we propose a simple modification to recover the original formulation of weight decay regularization by decoupling the weight decay from the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). Our proposed decoupled weight decay has already been adopted by many researchers, and the community has implemented it in TensorFlow and PyTorch; the complete source code for our experiments is available at https://github.com/loshchil/AdamW-and-SGDW

Key facts. Cited by 1,722 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 152; 2022: 234; 2023: 261; 2024: 427; 2025: 385; 2026: 262.

1,722 papers.

Sort:

By area:

  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  9. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  10. 2021Multiscale Vision Transformerscited by 896
  11. 2021Going deeper with Image Transformerscited by 759
  12. 2022Galactica: A Large Language Model for Sciencecited by 736
38 more
  1. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  2. 2024DeepSeek-V3 Technical Reportcited by 724
  3. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  4. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  5. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  6. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  7. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  8. 2023The Falcon Series of Open Language Modelscited by 526
  9. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  10. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  11. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  12. 2021TransReID: Transformer-based Object Re-Identificationcited by 432
  13. 2021Conditional DETR for Fast Training Convergencecited by 422
  14. 2022DeiT III: Revenge of the ViTcited by 417
  15. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  16. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  17. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  18. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 356
  19. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  20. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 325
  21. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  22. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 294
  23. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  24. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  25. 2021PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformerscited by 284
  26. 2022VectorMapNet: End-to-end Vectorized HD Map Learningcited by 243
  27. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  28. 2023MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusioncited by 210
  29. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  30. 2023SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagerycited by 202
  31. 2022Patches Are All You Need?cited by 201
  32. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  33. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  34. 2024Poseidon: Efficient Foundation Models for PDEscited by 186
  35. 2023Efficient Large Language Models: A Surveycited by 183
  36. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
  37. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  38. 2022Towards Metrical Reconstruction of Human Facescited by 165

50 most cited of 1,722 All 1,722 (JSON)