mlsanityData to 14 Oct 2026

A Universal Law of Robustness via Isoperimetry

arXiv 2105.12806 ↗First posted 26 May 2021cs.LG stat.MLcited by 151

Not on the map. Why?

Abstract (arXiv)

Classically, data interpolation with a parametrized model class is possible as long as the number of parameters is larger than the number of equations to be satisfied. A puzzling phenomenon in deep learning is that models are trained with many more parameters than what this classical theory would suggest. We propose a partial theoretical explanation for this phenomenon. We prove that for a broad class of data distributions and model classes, overparametrization is necessary if one wants to interpolate the data smoothly. Namely we show that smooth interpolation requires d times more parameters than mere interpolation, where d is the ambient data dimension. We prove this universal law of robustness for any smoothly parametrized function class with polynomial size weights, and any covariate distribution verifying isoperimetry. In the case of two-layers neural networks and Gaussian covariates, this law was conjectured in prior work by Bubeck, Li and Nagaraj. We also give an interpretation of our result as an improved generalization bound for model classes consisting of smooth functions.

Key facts. Cited by 151 papers in the map (data through 14 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 26 May 2021.

Builds on

22 of 43 references.

References · 22

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  4. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  5. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  6. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  7. 2019Benign Overfitting in Linear Regressioncited by 549
  8. 2020Overfitting in adversarially robust deep learningcited by 506
  9. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  10. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 315
  11. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  12. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 293
10 more
  1. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 278
  2. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 250
  3. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 189
  4. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 182
  5. 2018Rademacher Complexity for Adversarially Robust Generalizationcited by 132
  6. 2018Small ReLU networks are powerful memorizers: a tight analysis of memorization capacitycited by 78
  7. 2019Convergence of Adversarial Training in Overparametrized Neural Networkscited by 58
  8. 2016Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflectionscited by 57
  9. 2020A law of robustness for two-layers neural networkscited by 55
  10. 2018On Computation and Generalization of GANs with Spectrum Controlcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 50; 2023: 43; 2024: 23; 2025: 13; 2026: 13.

151 papers.

Sort:

By area:

  1. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  2. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  3. 2023Cross-Entropy Loss Functions: Theoretical Analysis and Applicationscited by 210
  4. 2023Understanding plasticity in neural networkscited by 174
  5. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 85
  6. 2023Direct Parameterization of Lipschitz-Bounded Deep Networkscited by 51
  7. 2021On the Optimal Memorization Power of ReLU Neural Networkscited by 40
  8. 2022Robust and Efficient Medical Imaging with Self-Supervisioncited by 40
  9. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 39
  10. 2022A Law of Data Separation in Deep Learningcited by 36
  11. 2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 34
  12. 2022On the influence of over-parameterization in manifold based surrogates and deep neural operatorscited by 29
38 more
  1. 2022On the Adversarial Robustness of Mixture of Expertscited by 28
  2. 2021Adversarial Examples in Multi-Layer Random ReLU Networkscited by 26
  3. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 26
  4. 2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25
  5. 2022SALTED: A Framework for SAlient Long-Tail Translation Error Detectioncited by 23
  6. 2022Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Powercited by 23
  7. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
  8. 2021Parameterizing Activation Functions for Adversarial Robustnesscited by 22
  9. 2022Overparameterized Linear Regression under Adversarial Attackscited by 21
  10. 2022Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)cited by 18
  11. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 18
  12. 2024Hardness of Learning Neural Networks under the Manifold Hypothesiscited by 18
  13. 2022Investigating Power laws in Deep Representation Learningcited by 17
  14. 2022DASHA: Distributed Nonconvex Optimization with Communication Compression, Optimal Oracle Complexity, and No Client Synchronizationcited by 16
  15. 2022A Quantitative Geometric Approach to Neural-Network Smoothnesscited by 16
  16. 2022Training Fully Connected Neural Networks is existsℝ-Completecited by 15
  17. 2022Evolution of Neural Tangent Kernels under Benign and Adversarial Trainingcited by 14
  18. 2023Unlocking Deterministic Robustness Certification on ImageNetcited by 13
  19. 2023Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernelscited by 13
  20. 2024Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Datacited by 12
  21. 2021On the Impact of Hard Adversarial Instances on Overfitting in Adversarial Trainingcited by 11
  22. 2022On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimescited by 11
  23. 2023On the Lipschitz Constant of Deep Networks and Double Descentcited by 11
  24. 2023How Spurious Features Are Memorized: Precise Analysis for Random and NTK Featurescited by 11
  25. 2022Towards Semantic Communication Protocols: A Probabilistic Logic Perspectivecited by 10
  26. 2023Data efficiency and extrapolation trends in neural network interatomic potentialscited by 10
  27. 2023Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networkscited by 10
  28. 2022A Law of Robustness beyond Isoperimetrycited by 9
  29. 2022Adversarially Robust Learning with Tolerancecited by 9
  30. 2022Sequential Attention for Feature Selectioncited by 9
  31. 2024The Surprising Harmfulness of Benign Overfitting for Adversarial Robustnesscited by 9
  32. 2025Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterizationcited by 9
  33. 2022Process Knowledge-infused Learning for Suicidality Assessment on Social Mediacited by 8
  34. 2022A law of adversarial risk, interpolation, and label noisecited by 8
  35. 2022Deep Double Descent via Smooth Interpolationcited by 8
  36. 2022Nash Equilibria and Pitfalls of Adversarial Training in Adversarial Robustness Gamescited by 8
  37. 2023Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Linear Subspacescited by 8
  38. 2024No Free Prune: Information-Theoretic Barriers to Pruning at Initializationcited by 8

50 most cited of 151 All 151 (JSON)