mlsanityData to 14 Oct 2026

Moment Centralization based Gradient Descent Optimizers for Convolutional Neural Networks

arXiv 2207.09066 ↗First posted 19 Jul 2022cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Convolutional neural networks (CNNs) have shown very appealing performance for many computer vision applications. The training of CNNs is generally performed using stochastic gradient descent (SGD) based optimization techniques. The adaptive momentum-based SGD optimizers are the recent trends. However, the existing optimizers are not able to maintain a zero mean in the first-order moment and struggle with optimization. In this paper, we propose a moment centralization-based SGD optimizer for CNNs. Specifically, we impose the zero mean constraints on the first-order moment explicitly. The proposed moment centralization is generic in nature and can be integrated with any of the existing adaptive momentum-based optimizers. The proposed idea is tested with three state-of-the-art optimization techniques, including Adam, Radam, and Adabelief on benchmark CIFAR10, CIFAR100, and TinyImageNet datasets for image classification. The performance of the existing optimizers is generally improved when integrated with the proposed moment centralization. Further, The results of the proposed moment centralization are also better than the existing gradient centralization. The analytical analysis using the toy example shows that the proposed method leads to a shorter and smoother optimization trajectory. The source code is made publicly available at urlhttps://github.com/sumanthsadhu/MC-optimizer.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 27 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

17 of 27 references.

References · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  6. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  7. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  8. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  9. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  10. 2020Gradient Centralization: A New Optimization Technique for Deep Neural Networkscited by 82
  11. 2020Meta Batch-Instance Normalization for Generalizable Person Re-Identificationcited by 77
  12. 2019diffGrad: An Optimization Method for Convolutional Neural Networkscited by 29
5 more
  1. 2018Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning ratecited by 16
  2. 2018RCCNet: An Efficient Convolutional Neural Network for Histological Routine Colon Cancer Nuclei Classificationcited by 11
  3. 2021AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networkscited by 9
  4. 2018Dual CNN Models for Unsupervised Monocular Depth Estimationcited by 8
  5. 2019Hard-Mining Loss based Convolutional Neural Network for Face Recognitioncited by 4

Built on by

Not cited yet.