mlsanityData to 7 Oct 2026

Enhancing Adversarial Training with Second-Order Statistics of Weights

arXiv 2203.06020 ↗First posted 11 Mar 2022cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Adversarial training has been shown to be one of the most effective approaches to improve the robustness of deep neural networks. It is formalized as a min-max optimization over model weights and adversarial perturbations, where the weights can be optimized through gradient descent methods like SGD. In this paper, we show that treating model weights as random variables allows for enhancing adversarial training through Second-Order Statistics Optimization (S2O) with respect to the weights. By relaxing a common (but unrealistic) assumption of previous PAC-Bayesian frameworks that all weights are statistically independent, we derive an improved PAC-Bayesian adversarial generalization bound, which suggests that optimizing second-order statistics of weights can effectively tighten the bound. In addition to this theoretical insight, we conduct an extensive set of experiments, which show that S2O not only improves the robustness and generalization of the trained neural networks when used in isolation, but also integrates easily in state-of-the-art adversarial training techniques like TRADES, AWP, MART, and AVMixup, leading to a measurable improvement of these techniques. The code is available at urlhttps://github.com/Alexkael/S2O.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 58 of its 86 references that are in the map. First posted on arXiv 11 Mar 2022.

Builds on

58 of 86 references.

References · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2016Wide Residual Networkscited by 3,057
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  10. 2016Adversarial examples in the physical worldcited by 1,762
  11. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  12. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
38 more
  1. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  3. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  4. 2019When Does Label Smoothing Help?cited by 985
  5. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  6. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  7. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  8. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  9. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  10. 2017Exploring Generalization in Deep Learningcited by 595
  11. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  12. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  13. 2020Overfitting in adversarially robust deep learningcited by 504
  14. 2017Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networkscited by 484
  15. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
  16. 2019Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attackcited by 333
  17. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  18. 2018Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionalitycited by 280
  19. 2018Adversarial Logit Pairingcited by 224
  20. 2020Bag of Tricks for Adversarial Trainingcited by 199
  21. 2020How Does Mixup Help With Robustness and Generalization?cited by 193
  22. 2018MMA Training: Direct Input Space Margin Maximization through Adversarial Trainingcited by 171
  23. 2020Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNetscited by 165
  24. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  25. 2015All you need is a good initcited by 124
  26. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  27. 2020Tighter risk certificates for neural networkscited by 112
  28. 2014Preserving Statistical Validity in Adaptive Data Analysiscited by 102
  29. 2019Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Trainingcited by 95
  30. 2020Learnable Boundary Guided Adversarial Trainingcited by 93
  31. 2017Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priorscited by 81
  32. 2020Adversarial Distributional Training for Robust Deep Learningcited by 79
  33. 2019Metric Learning for Adversarial Robustnesscited by 77
  34. 2015Generalization in Adaptive Data Analysis and Holdout Reusecited by 76
  35. 2020Adversarial Vertex Mixup: Toward Better Adversarially Robust Generalizationcited by 73
  36. 2018Generalizable Adversarial Training via Spectral Normalizationcited by 68
  37. 2016A Strongly Quasiconvex PAC-Bayesian Boundcited by 54
  38. 2020Second-Order Provable Defenses against Adversarial Attackscited by 39

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 10; 2024: 14; 2025: 7; 2026: 7.

42 papers.

Sort:

By area:

  1. 2023Cross-Entropy Loss Functions: Theoretical Analysis and Applicationscited by 206
  2. 2022SAFARI: Versatile and Efficient Evaluations for Robustness of Interpretabilitycited by 28
  3. 2023Randomized Adversarial Training via Taylor Expansioncited by 28
  4. 2023Feature Separation and Recalibration for Adversarial Robustnesscited by 16
  5. 2025Enhancing Robust Fairness via Confusional Spectral Regularizationcited by 11
  6. 2024Towards Fairness-Aware Adversarial Learningcited by 10
  7. 2023TrajPAC: Towards Robustness Verification of Pedestrian Trajectory Prediction Modelscited by 8
  8. 2022Certified Policy Smoothing for Cooperative Multi-Agent Reinforcement Learningcited by 7
  9. 2024Boosting Adversarial Training via Fisher-Rao Norm-based Regularizationcited by 7
  10. 2022Improving Robust Generalization by Direct PAC-Bayesian Bound Minimizationcited by 6
  11. 2023Adversarial Finetuning with Latent Representation Constraint to Mitigate Accuracy-Robustness Tradeoffcited by 6
  12. 2023Reward Certification for Policy Smoothed Reinforcement Learningcited by 6
30 more
  1. 2024Adversarial Training: A Surveycited by 6
  2. 2025CluCERT: Certifying LLM Robustness via Clustering-Guided Denoising Smoothingcited by 6
  3. 20223DVerifier: Efficient Robustness Verification for 3D Point Cloud Modelscited by 5
  4. 2023Indirect Gradient Matching for Adversarial Robust Distillationcited by 5
  5. 2024Adversarial Feature Alignment: Balancing Robustness and Accuracy in Deep Learning via Adversarial Trainingcited by 5
  6. 2024Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-offcited by 5
  7. 2025Long-tailed Adversarial Training with Self-Distillationcited by 5
  8. 2023Exploring Robust Features for Improving Adversarial Robustnesscited by 4
  9. 2024Catastrophic Overfitting: A Potential Blessing in Disguisecited by 4
  10. 2026Towards A Unified PAC-Bayesian Framework for Norm-based Generalization Boundscited by 4
  11. 2023Generating Less Certain Adversarial Examples Improves Robust Generalizationcited by 3
  12. 2024Out-of-Bounding-Box Triggers: A Stealthy Approach to Cheat Object Detectorscited by 3
  13. 2024Scene-Aware Explainable Multimodal Trajectory Predictioncited by 2
  14. 2025Shapley Uncertainty in Natural Language Generationcited by 2
  15. 2025Reconcile Certified Robustness and Accuracy for DNN-based Smoothed Majority Vote Classifiercited by 2
  16. 2025Test-Time Defense Against Adversarial Attacks via Stochastic Resonance of Latent Ensemblescited by 2
  17. 2024ZeroPur: Succinct Training-Free Adversarial Purificationcited by 1
  18. 2024Adversarial Robustness Overestimation and Instability in TRADEScited by 1
  19. 2026Margin-Adaptive Confidence Ranking for Reliable LLM Judgementcited by 1
  20. 2023Enhancing Robust Representation in Adversarial Training: Alignment and Exclusion Criteriacited by 0
  21. 2024Ensemble Adversarial Defense via Integration of Multiple Dispersed Low Curvature Modelscited by 0
  22. 2024Tightening the Evaluation of PAC Bounds Using Formal Verification Resultscited by 0
  23. 2024Data Augmentation for Continual RL via Adversarial Gradient Episodic Memorycited by 0
  24. 2024How Learning Dynamics Drive Adversarially Robust Generalization?cited by 0
  25. 2025Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillationcited by 0
  26. 2026S2O: Enhancing Adversarial Training with Second-Order Statistics of Weightscited by 0
  27. 2026Unveiling the Backdoor Mechanism Hidden Behind Catastrophic Overfitting in Fast Adversarial Trainingcited by 0
  28. 2026TrajRS: Towards Certified Robustness in Pedestrian Trajectory Predictioncited by 0
  29. 2026Localize-Then-Decide Guarantees for LLM Judgmentscited by 0
  30. 2026Interpretable but Fragile? Robustness of Concept Bottlenecks under Geometric-Semantic Perturbationscited by 0