# Gradient Amplification: An efficient way to train deep neural networks

arXiv: https://arxiv.org/abs/2006.10560 · HTML: https://mlsanity.com/paper/2006.10560/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 15 of 15)

- [Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs](https://mlsanity.com/paper/2107.13490/) (2021)
- [Collapse of Self-trained Language Models](https://mlsanity.com/paper/2404.02305/) (2024)
- [Stability-Informed Initialization of Neural Ordinary Differential Equations](https://mlsanity.com/paper/2311.15890/) (2023)
- [Differentiable modeling to unify machine learning and physical models and advance Geosciences](https://mlsanity.com/paper/2301.04027/) (2023)
- [Predicting the Emergence of Solar Active Regions Using Machine Learning](https://mlsanity.com/paper/2402.08890/) (2024)
- [MGTUNet: An new UNet for colon nuclei instance segmentation and quantification](https://mlsanity.com/paper/2210.10981/) (2022)
- [Informer in Algorithmic Investment Strategies on High Frequency Bitcoin Data](https://mlsanity.com/paper/2503.18096/) (2025)
- [How to Achieve the Intended Aim of Deep Clustering Now, without Deep Learning](https://mlsanity.com/paper/2602.05749/) (2026)
- [Covid-19 Detection from Chest X-ray and Patient Metadata using Graph Convolutional Neural Networks](https://mlsanity.com/paper/2105.09720/) (2021)
- [FEDI: Few-shot learning based on Earth Mover's Distance algorithm combined with deep residual network to identify diabetic retinopathy](https://mlsanity.com/paper/2108.09711/) (2021)
- [Dynamic Neural Network Architectural and Topological Adaptation and Related Methods -- A Survey](https://mlsanity.com/paper/2108.10066/) (2021)
- [AI enhanced finite element multiscale modelling and structural uncertainty analysis of a functionally graded porous beam](https://mlsanity.com/paper/2211.01970/) (2022)
- [SING: A Plug-and-Play DNN Learning Technique](https://mlsanity.com/paper/2305.15997/) (2023)
- [Intelligent gradient amplification for deep neural networks](https://mlsanity.com/paper/2305.18445/) (2023)
- [Value Prediction for Spatiotemporal Gait Data Using Deep Learning](https://mlsanity.com/paper/2403.07926/) (2024)

## Abstract (arXiv)

Improving performance of deep learning models and reducing their training times are ongoing challenges in deep neural networks. There are several approaches proposed to address these challenges one of which is to increase the depth of the neural networks. Such deeper networks not only increase training times, but also suffer from vanishing gradients problem while training. In this work, we propose gradient amplification approach for training deep learning models to prevent vanishing gradients and also develop a training strategy to enable or disable gradient amplification method across several epochs with different learning rates. We perform experiments on VGG-19 and resnet (Resnet-18 and Resnet-34) models, and study the impact of amplification parameters on these models in detail. Our proposed approach improves performance of these deep learning models even at higher learning rates, thereby allowing these models to achieve higher performance with reduced training time.