mlsanityData to 7 Oct 2026

Unintended Effects on Adaptive Learning Rate for Training Neural Network with Output Scale Change

arXiv 2103.03466 ↗First posted 5 Mar 2021cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

A multiplicative constant scaling factor is often applied to the model output to adjust the dynamics of neural network parameters. This has been used as one of the key interventions in an empirical study of lazy and active behavior. However, we show that the combination of such scaling and a commonly used adaptive learning rate optimizer strongly affects the training behavior of the neural network. This is problematic as it can cause unintended behavior of neural networks, resulting in the misinterpretation of experimental results. Specifically, for some scaling settings, the effect of the adaptive learning rate disappears or is strongly influenced by the scaling factor. To avoid the unintended effect, we present a modification of an optimization algorithm and demonstrate remarkable differences between adaptive learning rate optimization and simple gradient descent, especially with a small (<1.0) scaling factor.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 5 Mar 2021.

Builds on

24 of 32 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  8. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  9. 2018On Lazy Training in Differentiable Programmingcited by 671
  10. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  12. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
12 more
  1. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  2. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  3. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  4. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  5. 2019Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernelscited by 160
  6. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  7. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  8. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  9. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  10. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  11. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  12. 2020Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTKcited by 18

Built on by

Not cited yet.