mlsanityData to 7 Oct 2026

Dynamics of Local Elasticity During Training of Neural Nets

arXiv 2111.01166 ↗First posted 1 Nov 2021cs.LG math.DS stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

In the recent past, a property of neural training trajectories in weight-space had been isolated, that of "local elasticity" (denoted as Srm rel). Local elasticity attempts to quantify the propagation of the influence of a sampled data point on the prediction at another data. In this work, we embark on a comprehensive study of the existing notion of Srm rel and also propose a new definition that addresses the limitations that we point out for the original definition in the classification setting. On various state-of-the-art neural network training on SVHN, CIFAR-10 and CIFAR-100 we demonstrate how our new proposal of Srm rel, as opposed to the original definition, much more sharply detects the property of the weight updates preferring to make prediction changes within the same class as the sampled data. In neural regression experiments we demonstrate that the original Srm rel reveals a 2-phase behavior -- that the training proceeds via an initial elastic phase when Srm rel changes rapidly and an eventual inelastic phase when Srm rel remains large. We show that some of these properties can be analytically reproduced in various instances of doing regression via gradient flows on model predictor classes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 1 Nov 2021.

Builds on

32 of 43 references.

References · 32

  1. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  4. 2017Deep Neural Networks as Gaussian Processescited by 617
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  6. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  8. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  10. 2017Learning to Plan Chemical Synthesescited by 175
  11. 2020Neural collapse with unconstrained featurescited by 140
  12. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
20 more
  1. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  2. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  3. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  4. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  5. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  6. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  7. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
  8. 2020Agnostic Learning of a Single Neuron with Gradient Descentcited by 46
  9. 2020Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimizationcited by 41
  10. 2020Toward Better Generalization Bounds with Locally Elastic Stabilitycited by 41
  11. 2019The Local Elasticity of Neural Networkscited by 37
  12. 2020Approximation Schemes for ReLU Regressioncited by 34
  13. 2019Time/Accuracy Tradeoffs for Learning a ReLU with respect to Gaussian Marginalscited by 32
  14. 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
  15. 2019Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Networkcited by 27
  16. 2018Learning One Convolutional Layer with Overlapping Patchescited by 21
  17. 2020Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticitycited by 18
  18. 2020Depth-2 Neural Networks Under a Data-Poisoning Attackcited by 6
  19. 2020Provable Training of a ReLU Gate with an Iterative Non-Gradient Algorithmcited by 3
  20. 2021A Study of the Mathematics of Deep Learningcited by 3

Built on by

Not cited yet.