mlsanityData to 7 Oct 2026

An overview of gradient descent optimization algorithms

arXiv 1609.04747 ↗First posted 15 Sep 2016cs.LGcited by 1,021

Not on the map. Why?

Abstract (arXiv)

Gradient descent optimization algorithms, while increasingly popular, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by. This article aims to provide the reader with intuitions with regard to the behaviour of different algorithms that will allow her to put them to use. In the course of this overview, we look at different variants of gradient descent, summarize challenges, introduce the most common optimization algorithms, review architectures in a parallel and distributed setting, and investigate additional strategies for optimizing gradient descent.

Key facts. Cited by 1,021 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Sep 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 181; 2022: 178; 2023: 206; 2024: 232; 2025: 164; 2026: 60.

1,021 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  3. 2022ADBench: Anomaly Detection Benchmarkcited by 230
  4. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  5. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  6. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  7. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  8. 2021Efficient training of physics-informed neural networks via importance samplingcited by 109
  9. 2022Narcissus: A Practical Clean-Label Backdoor Attack with Limited Informationcited by 102
  10. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  11. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  12. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
38 more
  1. 2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
  2. 2022Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protectioncited by 80
  3. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  4. 2021RobustART: Benchmarking Robustness on Architecture Design and Training Techniquescited by 78
  5. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 78
  6. 2022Data-free Backdoor Removal based on Channel Lipschitznesscited by 77
  7. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  8. 2021Unsupervised MRI Reconstruction via Zero-Shot Learned Adversarial Transformerscited by 76
  9. 2024Optimal Flow Matching: Learning Straight Trajectories in Just One Stepcited by 74
  10. 2022Fair Contrastive Learning for Facial Attribute Classificationcited by 71
  11. 2022Formal Algorithms for Transformerscited by 70
  12. 2023AdaPoinTr: Diverse Point Cloud Completion with Adaptive Geometry-Aware Transformerscited by 65
  13. 2023GenerateCT: Text-Conditional Generation of 3D Chest CT Volumescited by 64
  14. 2023HeadGaS: Real-Time Animatable Head Avatars via 3D Gaussian Splattingcited by 60
  15. 2022Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Frameworkcited by 59
  16. 2021Quantum Architecture Search via Deep Reinforcement Learningcited by 58
  17. 2022A Review of Deep Transfer Learning and Recent Advancementscited by 56
  18. 2022GaitEdge: Beyond Plain End-to-end Gait Recognition for Better Practicalitycited by 52
  19. 2021Knowledge-Preserving Incremental Social Event Detection via Heterogeneous GNNscited by 51
  20. 2021Layerwise Optimization by Gradient Decomposition for Continual Learningcited by 51
  21. 2025FLEX: Continuous Agent Evolution via Forward Learning from Experiencecited by 51
  22. 2021Neighbor-view Enhanced Model for Vision and Language Navigationcited by 50
  23. 2021Efficient Visual Tracking with Exemplar Transformerscited by 48
  24. 2024Offline Training of Language Model Agents with Functions as Learnable Weightscited by 47
  25. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 47
  26. 2026AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimizationcited by 46
  27. 2023Grad-PU: Arbitrary-Scale Point Cloud Upsampling via Gradient Descent with Learned Distance Functionscited by 43
  28. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  29. 2023Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Datacited by 40
  30. 2023Towards Open-Set Test-Time Adaptation Utilizing the Wisdom of Crowds in Entropy Minimizationcited by 39
  31. 2024Contrastive Mean-Shift Learning for Generalized Category Discoverycited by 39
  32. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 38
  33. 2024LLM4Decompile: Decompiling Binary Code with Large Language Modelscited by 38
  34. 2021Momentum Residual Neural Networkscited by 36
  35. 2021Category Theory in Machine Learningcited by 36
  36. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  37. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learningcited by 33
  38. 2023Correcting model misspecification in physics-informed neural networks (PINNs)cited by 33

50 most cited of 1,021 All 1,021 (JSON)