# The Local Elasticity of Neural Networks

arXiv: https://arxiv.org/abs/1910.06943 · HTML: https://mlsanity.com/paper/1910.06943/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2019.

## Built on by (most cited first, 30 of 37)

- [Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02934/) (2021)
- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Diverse Weight Averaging for Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.09739/) (2022)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [Learning Dynamics of LLM Finetuning](https://mlsanity.com/paper/2407.10490/) (2024)
- [The alignment property of SGD noise and how it helps select flat minima: A stability analysis](https://mlsanity.com/paper/2207.02628/) (2022)
- [Semi-Supervised Active Learning with Temporal Output Discrepancy](https://mlsanity.com/paper/2107.14153/) (2021)
- [The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descent](https://mlsanity.com/paper/2305.17490/) (2023)
- [A Law of Data Separation in Deep Learning](https://mlsanity.com/paper/2210.17020/) (2022)
- [On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization](https://mlsanity.com/paper/2505.18830/) (2025)
- [On the Generalization Mystery in Deep Learning](https://mlsanity.com/paper/2203.10036/) (2022)
- [Neural (Tangent Kernel) Collapse](https://mlsanity.com/paper/2305.16427/) (2023)
- [The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View](https://mlsanity.com/paper/2510.04028/) (2025)
- [TiDAL: Learning Training Dynamics for Active Learning](https://mlsanity.com/paper/2210.06788/) (2022)
- [Elephant Neural Networks: Born to Be a Continual Learner](https://mlsanity.com/paper/2310.01365/) (2023)
- [When Representations Align: Universality in Representation Learning Dynamics](https://mlsanity.com/paper/2402.09142/) (2024)
- [Disparity Between Batches as a Signal for Early Stopping](https://mlsanity.com/paper/2107.06665/) (2021)
- [Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations](https://mlsanity.com/paper/2110.05960/) (2021)
- [Understanding Deep Learning via Decision Boundary](https://mlsanity.com/paper/2206.01515/) (2022)
- [Low-Dimensional Gradient Helps Out-of-Distribution Detection](https://mlsanity.com/paper/2310.17163/) (2023)
- [Global Convergence of SGD On Two Layer Neural Nets](https://mlsanity.com/paper/2210.11452/) (2022)
- [Understanding Simplicity Bias towards Compositional Mappings via Learning Dynamics](https://mlsanity.com/paper/2409.09626/) (2024)
- [On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity](https://mlsanity.com/paper/2205.12642/) (2022)
- [Envisioning Future Deep Learning Theories: Some Basic Concepts and Characteristics](https://mlsanity.com/paper/2112.09741/) (2021)
- [Global Convergence of SGD For Logistic Loss on Two Layer Neural Nets](https://mlsanity.com/paper/2309.09258/) (2023)
- [How Graph Neural Networks Learn: Lessons from Training Dynamics](https://mlsanity.com/paper/2310.05105/) (2023)
- [LEAD: Exploring Logit Space Evolution for Model Selection](https://mlsanity.com/paper/2507.14559/) (2025)
- [Beyond Reward Margin: Rethinking and Resolving Likelihood Displacement in Diffusion Models via Video Generation](https://mlsanity.com/paper/2511.19049/) (2025)
- [Temporal Output Discrepancy for Loss Estimation-based Active Learning](https://mlsanity.com/paper/2212.10613/) (2022)
- [Class-wise Generalization Error: an Information-Theoretic Analysis](https://mlsanity.com/paper/2401.02904/) (2024)

## Abstract (arXiv)

This paper presents a phenomenon in neural networks that we refer to as textitlocal elasticity. Roughly speaking, a classifier is said to be locally elastic if its prediction at a feature vector bx' is textitnot significantly perturbed, after the classifier is updated via stochastic gradient descent at a (labeled) feature vector bx that is textitdissimilar to bx' in a certain sense. This phenomenon is shown to persist for neural networks with nonlinear activation functions through extensive simulations on real-life and synthetic datasets, whereas this is not observed in linear classifiers. In addition, we offer a geometric interpretation of local elasticity using the neural tangent kernel citepjacot2018neural. Building on top of local elasticity, we obtain pairwise similarity measures between feature vectors, which can be used for clustering in conjunction with K-means. The effectiveness of the clustering algorithm on the MNIST and CIFAR-10 datasets in turn corroborates the hypothesis of local elasticity of neural networks on real-life data. Finally, we discuss some implications of local elasticity to shed light on several intriguing aspects of deep neural networks.