mlsanityData to 7 Oct 2026

Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)

arXiv 2209.07263 ↗First posted 15 Sep 2022cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

We study the average robustness notion in deep neural networks in (selected) wide and narrow, deep and shallow, as well as lazy and non-lazy training settings. We prove that in the under-parameterized setting, width has a negative effect while it improves robustness in the over-parameterized setting. The effect of depth closely depends on the initialization and the training mode. In particular, when initialized with LeCun initialization, depth helps robustness with the lazy training regime. In contrast, when initialized with Neural Tangent Kernel (NTK) and He-initialization, depth hurts the robustness. Moreover, under the non-lazy training regime, we demonstrate how the width of a two-layer ReLU network benefits robustness. Our theoretical developments improve the results by [Huang et al. NeurIPS21; Wu et al. NeurIPS21] and are consistent with [Bubeck and Sellke NeurIPS21; Bubeck et al. COLT21].

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 47 of its 64 references that are in the map. First posted on arXiv 15 Sep 2022.

Builds on

47 of 64 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  9. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  10. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  11. 2018On Lazy Training in Differentiable Programmingcited by 671
  12. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
35 more
  1. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  2. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  3. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  4. 2018Adversarially Robust Generalization Requires More Datacited by 318
  5. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  6. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  7. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  8. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  9. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 182
  10. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 181
  11. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  12. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  13. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  14. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  15. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  16. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  17. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  18. 2020Do Wider Neural Networks Really Help Adversarial Robustness?cited by 79
  19. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  20. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  21. 2021Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networkscited by 73
  22. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  23. 2019Convergence of Adversarial Training in Overparametrized Neural Networkscited by 58
  24. 2018Gradient Descent Quantizes ReLU Network Featurescited by 57
  25. 2020A law of robustness for two-layers neural networkscited by 54
  26. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  27. 2022The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regressioncited by 40
  28. 2020Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionalitycited by 36
  29. 2022Probabilistically Robust Learning: Balancing Average- and Worst-case Performancecited by 36
  30. 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
  31. 2020Robust Recovery via Implicit Bias of Discrepant Learning Rates for Double Over-parameterizationcited by 26
  32. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  33. 2022On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimescited by 10
  34. 2019An Analysis of the Expressiveness of Deep Neural Network Architectures Based on Their Lipschitz Constantscited by 5
  35. 2020Understanding the Intrinsic Robustness of Image Distributions using Conditional Generative Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 6; 2025: 1; 2026: 3.

18 papers.

Sort:

  1. 2022Revisiting Residual Networks for Adversarial Robustness: An Architectural Perspectivecited by 28
  2. 2023Understanding Certified Training with Interval Bound Propagationcited by 26
  3. 2023Sample Complexity Bounds for Score-Matching: Causal Discovery and Generative Modelingcited by 15
  4. 2023Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernelscited by 13
  5. 2024GlucoBench: Curated List of Continuous Glucose Monitoring Datasets with Prediction Benchmarkscited by 11
  6. 2023Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networkscited by 9
  7. 2023Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approachcited by 8
  8. 2024Robust NAS under adversarial training: benchmark, theory, and beyondcited by 6
  9. 2024A direct proof of a unified law of robustness for Bregman divergence lossescited by 4
  10. 2024Over-parameterization and Adversarial Robustness in Neural Networks: An Overview and Empirical Analysiscited by 2
  11. 2023Intrinsic Biologically Plausible Adversarial Robustnesscited by 1
  12. 2024Sparse and Transferable Universal Singular Vectors Attackcited by 1
6 more
  1. 2023Having Second Thoughts? Let's hear itcited by 0
  2. 2024Coordinated Sparse Recovery of Label Noisecited by 0
  3. 2025Approach to Finding a Robust Deep Learning Modelcited by 0
  4. 2026The Price of Robustness: Stable Classifiers Need Overparameterizationcited by 0
  5. 2026Precise Verification of Transformers through ReLU-Catalyzed Abstraction Refinementcited by 0
  6. 2026When Interpretability Becomes a Liability: Adversarial Attacks on CBM Concept Layerscited by 0