# diffGrad: An Optimization Method for Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1909.11015 · HTML: https://mlsanity.com/paper/1909.11015/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2019.

## Built on by (most cited first, 29 of 29)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Multimodal Fusion Transformer for Remote Sensing Image Classification](https://mlsanity.com/paper/2203.16952/) (2022)
- [Hyperspectral Image Classification-Traditional to Deep Models: A Survey for Future Prospects](https://mlsanity.com/paper/2101.06116/) (2021)
- [A Comprehensive Survey for Hyperspectral Image Classification: The Evolution from Conventional to Transformers and Mamba Models](https://mlsanity.com/paper/2404.14955/) (2024)
- [On the Power-Law Hessian Spectrums in Deep Learning](https://mlsanity.com/paper/2201.13011/) (2022)
- [AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks](https://mlsanity.com/paper/2105.10190/) (2021)
- [GeoFill: Reference-Based Image Inpainting with Better Geometric Understanding](https://mlsanity.com/paper/2201.08131/) (2022)
- [Training Recurrent Neural Networks by Sequential Least Squares and the Alternating Direction Method of Multipliers](https://mlsanity.com/paper/2112.15348/) (2021)
- [KOALA: A Kalman Optimization Algorithm with Loss Adaptivity](https://mlsanity.com/paper/2107.03331/) (2021)
- [AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs](https://mlsanity.com/paper/2210.06364/) (2022)
- [Exploiting Adam-like Optimization Algorithms to Improve the Performance of Convolutional Neural Networks](https://mlsanity.com/paper/2103.14689/) (2021)
- [High performing ensemble of convolutional neural networks for insect pest image detection](https://mlsanity.com/paper/2108.12539/) (2021)
- [Frequency Disentangled Residual Network](https://mlsanity.com/paper/2109.12556/) (2021)
- [AdaInject: Injection Based Adaptive Gradient Descent Optimizers for Convolutional Neural Networks](https://mlsanity.com/paper/2109.12504/) (2021)
- [Learning Speaker Representation with Semi-supervised Learning approach for Speaker Profiling](https://mlsanity.com/paper/2110.13653/) (2021)
- [Angle based dynamic learning rate for gradient descent](https://mlsanity.com/paper/2304.10457/) (2023)
- [Understanding Short-Range Memory Effects in Deep Neural Networks](https://mlsanity.com/paper/2105.02062/) (2021)
- [Joint Triplet Autoencoder for Histopathological Colon Cancer Nuclei Retrieval](https://mlsanity.com/paper/2105.10262/) (2021)
- [Semantic Video Segmentation for Intracytoplasmic Sperm Injection Procedures](https://mlsanity.com/paper/2101.01207/) (2021)
- [Painless step size adaptation for SGD](https://mlsanity.com/paper/2102.00853/) (2021)
- [AC-CovidNet: Attention Guided Contrastive CNN for Recognition of Covid-19 in Chest X-Ray Images](https://mlsanity.com/paper/2105.10239/) (2021)
- [Gated recurrent units and temporal convolutional network for multilabel classification](https://mlsanity.com/paper/2110.04414/) (2021)
- [Moment Centralization based Gradient Descent Optimizers for Convolutional Neural Networks](https://mlsanity.com/paper/2207.09066/) (2022)
- [Judging Adam: Studying the Performance of Optimization Methods on ML4SE Tasks](https://mlsanity.com/paper/2303.03540/) (2023)
- [Fine-Tuning Adaptive Stochastic Optimizers: Determining the Optimal Hyperparameter ε via Gradient Magnitude Histogram Analysis](https://mlsanity.com/paper/2311.11532/) (2023)
- [Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Tricks](https://mlsanity.com/paper/2503.02147/) (2025)
- [An Improved Adaptive PID Optimizer with Enhanced Convergence and Stability for Deep Learning](https://mlsanity.com/paper/2605.21968/) (2026)
- [AURA: Angular Update Rate Adaptation for training complex-valued neural networks](https://mlsanity.com/paper/2609.25914/) (2026)
- [EvE: An Alternate Optimizer to Adam](https://mlsanity.com/paper/2609.35614/) (2026)

## Abstract (arXiv)

Stochastic Gradient Decent (SGD) is one of the core techniques behind the success of deep neural networks. The gradient provides information on the direction in which a function has the steepest rate of change. The main problem with basic SGD is to change by equal sized steps for all parameters, irrespective of gradient behavior. Hence, an efficient way of deep network optimization is to make adaptive step sizes for each parameter. Recently, several attempts have been made to improve gradient descent methods such as AdaGrad, AdaDelta, RMSProp and Adam. These methods rely on the square roots of exponential moving averages of squared past gradients. Thus, these methods do not take advantage of local change in gradients. In this paper, a novel optimizer is proposed based on the difference between the present and the immediate past gradient (i.e., diffGrad). In the proposed diffGrad optimization technique, the step size is adjusted for each parameter in such a way that it should have a larger step size for faster gradient changing parameters and a lower step size for lower gradient changing parameters. The convergence analysis is done using the regret bound approach of online learning framework. Rigorous analysis is made in this paper over three synthetic complex non-convex functions. The image categorization experiments are also conducted over the CIFAR10 and CIFAR100 datasets to observe the performance of diffGrad with respect to the state-of-the-art optimizers such as SGDM, AdaGrad, AdaDelta, RMSProp, AMSGrad, and Adam. The residual unit (ResNet) based Convolutional Neural Networks (CNN) architecture is used in the experiments. The experiments show that diffGrad outperforms other optimizers. Also, we show that diffGrad performs uniformly well for training CNN using different activation functions. The source code is made publicly available at https://github.com/shivram1987/diffGrad.