mlsanityData to 7 Oct 2026

Activated Gradients for Deep Neural Networks

arXiv 2107.04228 ↗First posted 9 Jul 2021cs.CV cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Deep neural networks often suffer from poor performance or even training failure due to the ill-conditioned problem, the vanishing/exploding gradient problem, and the saddle point problem. In this paper, a novel method by acting the gradient activation function (GAF) on the gradient is proposed to handle these challenges. Intuitively, the GAF enlarges the tiny gradients and restricts the large gradient. Theoretically, this paper gives conditions that the GAF needs to meet, and on this basis, proves that the GAF alleviates the problems mentioned above. In addition, this paper proves that the convergence rate of SGD with the GAF is faster than that without the GAF under some assumptions. Furthermore, experiments on CIFAR, ImageNet, and PASCAL visual object classes confirm the GAF's effectiveness. The experimental results also demonstrate that the proposed method is able to be adopted in various deep neural networks to improve their performance. The source code is publicly available at https://github.com/LongJin-lab/Activated-Gradients-for-Deep-Neural-Networks.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 34 of its 61 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

34 of 61 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  9. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  11. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  12. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
22 more
  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  2. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  3. 2016An overview of gradient descent optimization algorithmscited by 1,021
  4. 2020Designing Network Design Spacescited by 954
  5. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  6. 2017Exploring Generalization in Deep Learningcited by 595
  7. 2017Self-Normalizing Neural Networkscited by 591
  8. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  9. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  10. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  11. 2017Dual Path Networkscited by 119
  12. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 89
  13. 2020Gradient Centralization: A New Optimization Technique for Deep Neural Networkscited by 82
  14. 2019Dependency-Guided LSTM-CRF for Named Entity Recognitioncited by 28
  15. 2018SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learningcited by 25
  16. 2017Accelerated Distributed Nesterov Gradient Descentcited by 24
  17. 2018Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalizationcited by 21
  18. 2017On the Effectiveness of Least Squares Generative Adversarial Networkscited by 18
  19. 2018L1-Norm Batch Normalization for Efficient Training of Deep Neural Networkscited by 15
  20. 2020A Study of Condition Numbers for First-Order Optimizationcited by 10
  21. 2016The role of convexity on saddle-point dynamics: Lyapunov function and robustnesscited by 5
  22. 2019The condition number of a function relative to a setcited by 1

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 2; 2023: 7; 2024: 2; 2025: 1.

14 papers.

Sort:

  1. 2024Exact Gauss-Newton Optimization for Training Deep Neural Networkscited by 10
  2. 2022Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networkscited by 5
  3. 2023Adversarial Attacks and Defenses in Machine Learning-Powered Networks: A Contemporary Surveycited by 5
  4. 2023A Dynamic Linear Bias Incorporation Scheme for Nonnegative Latent Factor Analysiscited by 3
  5. 2023Proximal Symmetric Non-negative Latent Factor Analysis: A Novel Approach to Highly-Accurate Representation of Undirected Weighted Networkscited by 2
  6. 2022Zero Stability Well Predicts Performance of Convolutional Neural Networkscited by 1
  7. 2024Residual resampling-based physics-informed neural network for neutron diffusion equationscited by 1
  8. 2025DeepFeatIoT: Unifying Deep Learned, Randomized, and LLM Features for Enhanced IoT Time Series Sensor Data Classification in Smart Industriescited by 1
  9. 2021Transformer-Encoder-GRU (T-E-GRU) for Chinese Sentiment Analysis on Chinese Comment Textcited by 0
  10. 2021Adaptive Zeroing-Type Neural Dynamics for Solving Quadratic Minimization and Applied to Target Trackingcited by 0
  11. 2023A Constraints Fusion-induced Symmetric Nonnegative Matrix Factorization Approach for Community Detectioncited by 0
  12. 2023Online Sparse Streaming Feature Selection Using Adapted Classificationcited by 0
2 more
  1. 2023An Incomplete Tensor Tucker decomposition based Traffic Speed Prediction Methodcited by 0
  2. 2023A Novel Method for improving accuracy in neural network by reinstating traditional back propagation techniquecited by 0