mlsanityData to 7 Oct 2026

Sufficient Conditions for Stability of Minimum-Norm Interpolating Deep ReLU Networks

arXiv 2602.13910 ↗First posted 14 Feb 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Algorithmic stability is a classical framework for analyzing the generalization error of learning algorithms. It predicts that an algorithm has small generalization error if it is insensitive to small perturbations in the training set such as the removal or replacement of a training point. While stability has been demonstrated for numerous well-known algorithms, this framework has had limited success in analyses of deep neural networks. In this paper we study the algorithmic stability of deep ReLU homogeneous neural networks that achieve zero training error using parameters with the smallest L2 norm, also known as the minimum-norm interpolation, a phenomenon that can be observed in overparameterized models trained by gradient-based algorithms. We investigate sufficient conditions for such networks to be stable. We find that 1) such networks are stable when they contain a (possibly small) stable sub-network, followed by a layer with a low-rank weight matrix, and 2) such networks are not guaranteed to be stable even when they contain a stable sub-network, if the following layer is not low-rank. The low-rank assumption is inspired by recent empirical and theoretical results which demonstrate that training deep neural networks is biased towards low-rank weight matrices, for minimum-norm interpolation and weight-decay regularization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 14 Feb 2026.

Builds on

33 of 50 references.

References · 33

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  3. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  4. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  5. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  6. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  7. 2019Benign Overfitting in Linear Regressioncited by 544
  8. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  10. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  11. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  12. 2020Directional convergence and alignment in deep learningcited by 172
21 more
  1. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  2. 2020Neural collapse with unconstrained featurescited by 140
  3. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  4. 2020Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descentcited by 118
  5. 2017Stability and Generalization of Learning Algorithms that Converge to Global Optimacited by 100
  6. 2019Sharper bounds for uniformly stable algorithmscited by 91
  7. 2017Data-Dependent Stability of Stochastic Gradient Descentcited by 90
  8. 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72
  9. 2020Logarithmic Pruning is All You Needcited by 71
  10. 2018Generalization Bounds for Uniformly Stable Algorithmscited by 56
  11. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  12. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  13. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  14. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  15. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  16. 2022Stability vs Implicit Bias of Gradient Methods on Separable Data and Beyondcited by 29
  17. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  18. 2021Time-independent Generalization Bounds for SGLD in Non-convex Settingscited by 27
  19. 2021Stability & Generalisation of Gradient Descent for Shallow Neural Networks without the Neural Tangent Kernelcited by 25
  20. 2021Learning with Gradient Descent and Weakly Convex Lossescited by 16
  21. 2022Analyzing Lottery Ticket Hypothesis from PAC-Bayesian Theory Perspectivecited by 9

Built on by

Not cited yet.