mlsanityData to 7 Oct 2026

Painless step size adaptation for SGD

arXiv 2102.00853 ↗First posted 1 Feb 2021cs.LG cs.NEcited by 0

Not on the map. Why?

Abstract (arXiv)

Convergence and generalization are two crucial aspects of performance in neural networks. When analyzed separately, these properties may lead to contradictory results. Optimizing a convergence rate yields fast training, but does not guarantee the best generalization error. To avoid the conflict, recent studies suggest adopting a moderately large step size for optimizers, but the added value on the performance remains unclear. We propose the LIGHT function with the four configurations which regulate explicitly an improvement in convergence and generalization on testing. This contribution allows to: 1) improve both convergence and generalization of neural networks with no need to guarantee their stability; 2) build more reliable and explainable network architectures with no need for overparameterization. We refer to it as "painless" step size adaptation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 111 references that are in the map. First posted on arXiv 1 Feb 2021.

Builds on

40 of 111 references.

References · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2017Exploring Generalization in Deep Learningcited by 595
  7. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  8. 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 403
  9. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  10. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  11. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  12. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
28 more
  1. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  2. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  3. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  4. 2016Deep Information Propagationcited by 194
  5. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  6. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
  7. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  8. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  9. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  10. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  11. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  12. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 87
  13. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  14. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  15. 2016Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regressioncited by 64
  16. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 63
  17. 2020Learning a Single Neuron with Gradient Methodscited by 50
  18. 2020Agnostic Learning of a Single Neuron with Gradient Descentcited by 46
  19. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  20. 2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
  21. 2019diffGrad: An Optimization Method for Convolutional Neural Networkscited by 29
  22. 2020Quantitative Propagation of Chaos for SGD in Wide Neural Networkscited by 20
  23. 2018Stochastic Learning under Random Reshuffling with Constant Step-sizescited by 14
  24. 2016Tradeoffs between Convergence Speed and Reconstruction Accuracy in Inverse Problemscited by 12
  25. 2020Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Ratecited by 12
  26. 2018Generalization and Expressivity for Deep Netscited by 11
  27. 2020Can Single Neurons Solve MNIST? The Computational Power of Biological Dendritic Treescited by 5
  28. 2018Breaking the Activation Function Bottleneck through Adaptive Parameterizationcited by 2

Built on by

Not cited yet.