# Gradient Centralization: A New Optimization Technique for Deep Neural Networks

arXiv: https://arxiv.org/abs/2004.01461 · HTML: https://mlsanity.com/paper/2004.01461/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2020.

## Built on by (most cited first, 30 of 82)

- [GDRNPP: A Geometry-guided and Fully Learning-based Object Pose Estimator](https://mlsanity.com/paper/2102.12145/) (2021)
- [GPV-Pose: Category-level Object Pose Estimation via Geometry-guided Point-wise Voting](https://mlsanity.com/paper/2203.07918/) (2022)
- [SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation](https://mlsanity.com/paper/2108.08367/) (2021)
- [Energy Transformer](https://mlsanity.com/paper/2302.07253/) (2023)
- [Hyperspectral Image Classification-Traditional to Deep Models: A Survey for Future Prospects](https://mlsanity.com/paper/2101.06116/) (2021)
- [HS-Pose: Hybrid Scope Feature Extraction for Category-level Object Pose Estimation](https://mlsanity.com/paper/2303.15743/) (2023)
- [RBP-Pose: Residual Bounding Box Projection for Category-Level Pose Estimation](https://mlsanity.com/paper/2208.00237/) (2022)
- [SSP-Pose: Symmetry-Aware Shape Prior Deformation for Direct Category-Level Object Pose Estimation](https://mlsanity.com/paper/2208.06661/) (2022)
- [PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Literature Parsing Task B: Table Recognition to HTML](https://mlsanity.com/paper/2105.01848/) (2021)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Occlusion-Aware Self-Supervised Monocular 6D Object Pose Estimation](https://mlsanity.com/paper/2203.10339/) (2022)
- [CATRE: Iterative Point Clouds Alignment for Category-level Object Pose Refinement](https://mlsanity.com/paper/2207.08082/) (2022)
- [Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning](https://mlsanity.com/paper/2106.05956/) (2021)
- [Ranger21: a synergistic deep learning optimizer](https://mlsanity.com/paper/2106.13731/) (2021)
- [A Comprehensive Survey for Hyperspectral Image Classification: The Evolution from Conventional to Transformers and Mamba Models](https://mlsanity.com/paper/2404.14955/) (2024)
- [Multilingual and cross-lingual document classification: A meta-learning approach](https://mlsanity.com/paper/2101.11302/) (2021)
- [Activated Gradients for Deep Neural Networks](https://mlsanity.com/paper/2107.04228/) (2021)
- [Average of Pruning: Improving Performance and Stability of Out-of-Distribution Detection](https://mlsanity.com/paper/2303.01201/) (2023)
- [LaPose: Laplacian Mixture Shape Modeling for RGB-Based Category-Level Object Pose Estimation](https://mlsanity.com/paper/2409.15727/) (2024)
- [OmniJet-αC: Learning point cloud calorimeter simulations using generative transformers](https://mlsanity.com/paper/2501.05534/) (2025)
- [TransNet: Category-Level Transparent Object Pose Estimation](https://mlsanity.com/paper/2208.10002/) (2022)
- [Understanding the Failure of Batch Normalization for Transformers in NLP](https://mlsanity.com/paper/2210.05153/) (2022)
- [PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Table Image Recognition to Latex](https://mlsanity.com/paper/2105.01846/) (2021)
- [CroMo: Cross-Modal Learning for Monocular Depth Estimation](https://mlsanity.com/paper/2203.12485/) (2022)
- [AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks](https://mlsanity.com/paper/2105.10190/) (2021)
- [ICDAR 2021 Competition on Scientific Table Image Recognition to LaTeX](https://mlsanity.com/paper/2105.14426/) (2021)
- [IntereStyle: Encoding an Interest Region for Robust StyleGAN Inversion](https://mlsanity.com/paper/2209.10811/) (2022)
- [GeoReF: Geometric Alignment Across Shape Variation for Category-level Object Pose Refinement](https://mlsanity.com/paper/2404.11139/) (2024)
- [Delta Sampling R-BERT for limited data and low-light action recognition](https://mlsanity.com/paper/2107.05202/) (2021)
- [Delving into the Estimation Shift of Batch Normalization in a Network](https://mlsanity.com/paper/2203.10778/) (2022)

## Abstract (arXiv)

Optimization techniques are of great importance to effectively and efficiently train a deep neural network (DNN). It has been shown that using the first and second order statistics (e.g., mean and variance) to perform Z-score standardization on network activations or weight vectors, such as batch normalization (BN) and weight standardization (WS), can improve the training performance. Different from these existing methods that mostly operate on activations or weights, we present a new optimization technique, namely gradient centralization (GC), which operates directly on gradients by centralizing the gradient vectors to have zero mean. GC can be viewed as a projected gradient descent method with a constrained loss function. We show that GC can regularize both the weight space and output feature space so that it can boost the generalization performance of DNNs. Moreover, GC improves the Lipschitzness of the loss function and its gradient so that the training process becomes more efficient and stable. GC is very simple to implement and can be easily embedded into existing gradient based DNN optimizers with only one line of code. It can also be directly used to fine-tune the pre-trained DNNs. Our experiments on various applications, including general image classification, fine-grained image classification, detection and segmentation, demonstrate that GC can consistently improve the performance of DNN learning. The code of GC can be found at https://github.com/Yonghongwei/Gradient-Centralization.