mlsanityData to 7 Oct 2026

A Comparison of Optimization Algorithms for Deep Learning

arXiv 2007.14166 ↗First posted 28 Jul 2020cs.LG stat.MLcited by 23

Not on the map. Why?

Abstract (arXiv)

In recent years, we have witnessed the rise of deep learning. Deep neural networks have proved their success in many areas. However, the optimization of these networks has become more difficult as neural networks going deeper and datasets becoming bigger. Therefore, more advanced optimization algorithms have been proposed over the past years. In this study, widely used optimization algorithms for deep learning are examined in detail. To this end, these algorithms called adaptive gradient methods are implemented for both supervised and unsupervised tasks. The behaviour of the algorithms during training and results on four image datasets, namely, MNIST, CIFAR-10, Kaggle Flowers and Labeled Faces in the Wild are compared by pointing out their differences against basic optimization algorithms.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 3; 2023: 3; 2024: 3; 2025: 8; 2026: 1.

23 papers.

Sort:

  1. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  2. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  3. 2021Prediction of lung and colon cancer through analysis of histopathological images by utilizing Pre-trained CNN models with visualization of class activation and saliency mapscited by 7
  4. 2021Bayesian brains and the R'enyi divergencecited by 6
  5. 2025Privacy Preserving Machine Learning Model Personalization through Federated Personalized Learningcited by 5
  6. 2025Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rulescited by 4
  7. 2022Stochastic Modeling of Inhomogeneities in the Aortic Wall and Uncertainty Quantification using a Bayesian Encoder-Decoder Surrogatecited by 2
  8. 2022Application of multilayer perceptron with data augmentation in nuclear physicscited by 1
  9. 2023Unveiling The Factors of Aesthetic Preferences with Explainable AIcited by 1
  10. 2024Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapescited by 1
  11. 2025HOTSPOT-YOLO: A Lightweight Deep Learning Attention-Driven Model for Detecting Thermal Anomalies in Drone-Based Solar Photovoltaic Inspectionscited by 1
  12. 2021Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learningcited by 0
11 more
  1. 2022Exploring the Optimized Value of Each Hyperparameter in Various Gradient Descent Algorithmscited by 0
  2. 2023Osteosarcoma Tumor Detection using Transfer Learning Modelscited by 0
  3. 2023ELSA: Partial Weight Freezing for Overhead-Free Sparse Network Deploymentcited by 0
  4. 2024Conjugate-Gradient-like Based Adaptive Moment Estimation Optimization Algorithm for Deep Learningcited by 0
  5. 2024Memory-Reduced Meta-Learning with Guaranteed Convergencecited by 0
  6. 2025Benchmarking Randomized Optimization Algorithms on Binary, Permutation, and Combinatorial Problem Landscapescited by 0
  7. 2025Exploration of Hepatitis B Virus Infection Dynamics through Physics-Informed Deep Learning Approachcited by 0
  8. 2025Deep Learning Models for Robust Facial Liveness Detectioncited by 0
  9. 2025Ontology-Aligned Embeddings for Data-Driven Labour Market Analyticscited by 0
  10. 2025Enhanced Chest Disease Classification Using an Improved CheXNet Framework with EfficientNetV2-M and Optimization-Driven Learningcited by 0
  11. 2026Novel Dynamic Batch-Sensitive Adam Optimiser for Vehicular Accident Injury Severity Predictioncited by 0