# A Comparison of Optimization Algorithms for Deep Learning

arXiv: https://arxiv.org/abs/2007.14166 · HTML: https://mlsanity.com/paper/2007.14166/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2020.

## Built on by (most cited first, 23 of 23)

- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect](https://mlsanity.com/paper/2110.03677/) (2021)
- [Prediction of lung and colon cancer through analysis of histopathological images by utilizing Pre-trained CNN models with visualization of class activation and saliency maps](https://mlsanity.com/paper/2103.12155/) (2021)
- [Bayesian brains and the R'enyi divergence](https://mlsanity.com/paper/2107.05438/) (2021)
- [Privacy Preserving Machine Learning Model Personalization through Federated Personalized Learning](https://mlsanity.com/paper/2505.01788/) (2025)
- [Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules](https://mlsanity.com/paper/2510.26997/) (2025)
- [Stochastic Modeling of Inhomogeneities in the Aortic Wall and Uncertainty Quantification using a Bayesian Encoder-Decoder Surrogate](https://mlsanity.com/paper/2202.10244/) (2022)
- [Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning](https://mlsanity.com/paper/2101.09192/) (2021)
- [Application of multilayer perceptron with data augmentation in nuclear physics](https://mlsanity.com/paper/2205.07953/) (2022)
- [Unveiling The Factors of Aesthetic Preferences with Explainable AI](https://mlsanity.com/paper/2311.14410/) (2023)
- [Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapes](https://mlsanity.com/paper/2409.11995/) (2024)
- [HOTSPOT-YOLO: A Lightweight Deep Learning Attention-Driven Model for Detecting Thermal Anomalies in Drone-Based Solar Photovoltaic Inspections](https://mlsanity.com/paper/2508.18912/) (2025)
- [Exploring the Optimized Value of Each Hyperparameter in Various Gradient Descent Algorithms](https://mlsanity.com/paper/2212.12279/) (2022)
- [Osteosarcoma Tumor Detection using Transfer Learning Models](https://mlsanity.com/paper/2305.09660/) (2023)
- [ELSA: Partial Weight Freezing for Overhead-Free Sparse Network Deployment](https://mlsanity.com/paper/2312.06872/) (2023)
- [Conjugate-Gradient-like Based Adaptive Moment Estimation Optimization Algorithm for Deep Learning](https://mlsanity.com/paper/2404.01714/) (2024)
- [Memory-Reduced Meta-Learning with Guaranteed Convergence](https://mlsanity.com/paper/2412.12030/) (2024)
- [Benchmarking Randomized Optimization Algorithms on Binary, Permutation, and Combinatorial Problem Landscapes](https://mlsanity.com/paper/2501.17170/) (2025)
- [Exploration of Hepatitis B Virus Infection Dynamics through Physics-Informed Deep Learning Approach](https://mlsanity.com/paper/2503.10708/) (2025)
- [Deep Learning Models for Robust Facial Liveness Detection](https://mlsanity.com/paper/2508.09094/) (2025)
- [Ontology-Aligned Embeddings for Data-Driven Labour Market Analytics](https://mlsanity.com/paper/2509.04942/) (2025)
- [Enhanced Chest Disease Classification Using an Improved CheXNet Framework with EfficientNetV2-M and Optimization-Driven Learning](https://mlsanity.com/paper/2512.08992/) (2025)
- [Novel Dynamic Batch-Sensitive Adam Optimiser for Vehicular Accident Injury Severity Prediction](https://mlsanity.com/paper/2605.15083/) (2026)

## Abstract (arXiv)

In recent years, we have witnessed the rise of deep learning. Deep neural networks have proved their success in many areas. However, the optimization of these networks has become more difficult as neural networks going deeper and datasets becoming bigger. Therefore, more advanced optimization algorithms have been proposed over the past years. In this study, widely used optimization algorithms for deep learning are examined in detail. To this end, these algorithms called adaptive gradient methods are implemented for both supervised and unsupervised tasks. The behaviour of the algorithms during training and results on four image datasets, namely, MNIST, CIFAR-10, Kaggle Flowers and Labeled Faces in the Wild are compared by pointing out their differences against basic optimization algorithms.