mlsanityData to 7 Oct 2026

Low Curvature Activations Reduce Overfitting in Adversarial Training

arXiv 2102.07861 ↗First posted 15 Feb 2021cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

Adversarial training is one of the most effective defenses against adversarial attacks. Previous works suggest that overfitting is a dominant phenomenon in adversarial training leading to a large generalization gap between test and train accuracy in neural networks. In this work, we show that the observed generalization gap is closely related to the choice of the activation function. In particular, we show that using activation functions with low (exact or approximate) curvature values has a regularization effect that significantly reduces both the standard and robust generalization gaps in adversarial training. We observe this effect for both differentiable/smooth activations such as SiLU as well as non-differentiable/non-smooth activations such as LeakyReLU. In the latter case, the "approximate" curvature of the activation is low. Finally, we show that for activation functions with low curvature, the double descent phenomenon for adversarially trained models does not occur.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 68 of its 96 references that are in the map. First posted on arXiv 15 Feb 2021.

Builds on

68 of 96 references.

References · 68

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2016Wide Residual Networkscited by 3,057
  8. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  9. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  10. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  11. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  12. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
38 more
  1. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  3. 2016Adversarial Machine Learning at Scalecited by 934
  4. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  5. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 739
  6. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  7. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  8. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  9. 2019Adversarial Training for Free!cited by 638
  10. 2017Exploring Generalization in Deep Learningcited by 595
  11. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 564
  12. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  13. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  14. 2017Countering Adversarial Images using Input Transformationscited by 517
  15. 2020Overfitting in adversarially robust deep learningcited by 504
  16. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 462
  17. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 439
  18. 2017Mitigating Adversarial Effects Through Randomizationcited by 382
  19. 2018On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Modelscited by 322
  20. 2018Adversarially Robust Generalization Requires More Datacited by 318
  21. 2018Feature Denoising for Improving Adversarial Robustnesscited by 312
  22. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  23. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  24. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 275
  25. 2016Safety Verification of Deep Neural Networkscited by 262
  26. 2017Detecting Adversarial Samples from Artifactscited by 244
  27. 2017PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examplescited by 243
  28. 2020Attacks Which Do Not Kill Training Make Adversarial Learning Strongercited by 242
  29. 2017Formal Verification of Piece-Wise Linear Feed-Forward Neural Networkscited by 241
  30. 2017On Detecting Adversarial Perturbationscited by 232
  31. 2018Sorting out Lipschitz function approximationcited by 211
  32. 2020Bag of Tricks for Adversarial Trainingcited by 199
  33. 2019You Only Propagate Once: Accelerating Adversarial Training via Maximal Principlecited by 181
  34. 2018Robustness via curvature regularization, and vice versacited by 174
  35. 2018Stochastic Activation Pruning for Robust Adversarial Defensecited by 171
  36. 2017Towards Robust Neural Networks via Random Self-ensemblecited by 158
  37. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 152
  38. 2018Scaling provable adversarial defensescited by 141

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 8; 2023: 5; 2024: 11; 2025: 1; 2026: 1.

29 papers.

Sort:

  1. 2022Robustness and Accuracy Could Be Reconcilable by (Proper) Definitioncited by 109
  2. 2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
  3. 2023Data Augmentation Alone Can Improve Adversarial Trainingcited by 43
  4. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  5. 2022Revisiting Residual Networks for Adversarial Robustness: An Architectural Perspectivecited by 28
  6. 2021Parameterizing Activation Functions for Adversarial Robustnesscited by 22
  7. 2024MeanSparse: Post-Training Robustness Enhancement Through Mean-Centered Feature Sparsificationcited by 20
  8. 2021Label Noise in Adversarial Training: A Novel Perspective to Study Robust Overfittingcited by 14
  9. 2022LinSyn: Synthesizing Tight Linear Bounds for Arbitrary Neural Network Activation Functionscited by 10
  10. 2024Shedding More Light on Robust Classifiers under the lens of Energy-based Modelscited by 9
  11. 2024Efficient local linearity regularization to overcome catastrophic overfittingcited by 7
  12. 2022Revisiting the Adversarial Robustness-Accuracy Tradeoff in Robot Learningcited by 6
17 more
  1. 2023A Simple Framework to Enhance the Adversarial Robustness of Deep Learning-based Intrusion Detection Systemcited by 6
  2. 2024Keeping Deep Learning Models in Check: A History-Based Approach to Mitigate Overfittingcited by 6
  3. 2024Characterizing Model Robustness via Natural Input Gradientscited by 6
  4. 2024Adversarial Training: A Surveycited by 6
  5. 2024Stability and Generalization in Free Adversarial Trainingcited by 5
  6. 2022Towards More Robust Interpretation via Local Gradient Alignmentcited by 4
  7. 2023Enhancing Adversarial Training via Reweighting Optimization Trajectorycited by 4
  8. 2022Fast Multi-grid Methods for Minimizing Curvature Energycited by 3
  9. 2024Compositional Curvature Bounds for Deep Neural Networkscited by 3
  10. 2022Bag of Tricks for FGSM Adversarial Trainingcited by 2
  11. 2025Understanding Adversarial Training with Energy-based Modelscited by 2
  12. 2024Feature compression is the root cause of adversarial fragility in neural network classifierscited by 1
  13. 2024Are Sparse Neural Networks Better Hard Sample Learners?cited by 1
  14. 2023On robust overfitting: adversarial training induced distribution matterscited by 0
  15. 2023Understanding the Role of Optimization in Double Descentcited by 0
  16. 2024Ensemble Adversarial Defense via Integration of Multiple Dispersed Low Curvature Modelscited by 0
  17. 2026A Composite Activation Function for Learning Stable Binary Representationscited by 0