mlsanityData to 7 Oct 2026

Neurons learn slower than they think

arXiv 2104.02578 ↗First posted 2 Apr 2021cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent studies revealed complex convergence dynamics in gradient-based methods, which has been little understood so far. Changing the step size to balance between high convergence rate and small generalization error may not be sufficient: maximizing the test accuracy usually requires a larger learning rate than minimizing the training loss. To explore the dynamic bounds of convergence rate, this study introduces textitdifferential capability into an optimization process, which measures whether the test accuracy increases as fast as a model approaches the decision boundary in a classification problem. The convergence analysis showed that: 1) a higher convergence rate leads to slower capability growth; 2) a lower convergence rate results in faster capability growth and decay; 3) regulating a convergence rate in either direction reduces differential capability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 2 Apr 2021.

Builds on

24 of 37 references.

References · 24

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  5. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  6. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  7. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  8. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
  9. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  10. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  11. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  12. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
12 more
  1. 2018Convergence of Gradient Descent on Separable Datacited by 108
  2. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 87
  3. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  4. 2016Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regressioncited by 64
  5. 2020Learning a Single Neuron with Gradient Methodscited by 50
  6. 2020Agnostic Learning of a Single Neuron with Gradient Descentcited by 46
  7. 2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
  8. 2020Quantitative Propagation of Chaos for SGD in Wide Neural Networkscited by 20
  9. 2019β3-IRT: A New Item Response Model and its Applicationscited by 17
  10. 2020Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Ratecited by 12
  11. 2018Generalization and Expressivity for Deep Netscited by 11
  12. 2020Can Single Neurons Solve MNIST? The Computational Power of Biological Dendritic Treescited by 5

Built on by

Not cited yet.