mlsanityData to 7 Oct 2026

Rethinking Adversarial Training with Neural Tangent Kernel

arXiv 2312.02236 ↗First posted 4 Dec 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Adversarial training (AT) is an important and attractive topic in deep learning security, exhibiting mysteries and odd properties. Recent studies of neural network training dynamics based on Neural Tangent Kernel (NTK) make it possible to reacquaint AT and deeply analyze its properties. In this paper, we perform an in-depth investigation of AT process and properties with NTK, such as NTK evolution. We uncover three new findings that are missed in previous works. First, we disclose the impact of data normalization on AT and the importance of unbiased estimators in batch normalization layers. Second, we experimentally explore the kernel dynamics and propose more time-saving AT methods. Third, we study the spectrum feature inside the kernel to address the catastrophic overfitting problem. To the best of our knowledge, it is the first work leveraging the observations of kernel dynamics to improve existing AT methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 32 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

25 of 32 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  9. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  10. 2016An overview of gradient descent optimization algorithmscited by 1,021
  11. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  12. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
13 more
  1. 2020Overfitting in adversarially robust deep learningcited by 504
  2. 2018Adversarially Robust Generalization Requires More Datacited by 318
  3. 2019Adversarial Examples Improve Image Recognitioncited by 283
  4. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  5. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  6. 2020Understanding Catastrophic Overfitting in Single-step Adversarial Trainingcited by 75
  7. 2021Exploring Memorization in Adversarial Trainingcited by 55
  8. 2021What can linearized neural networks actually say about generalization?cited by 49
  9. 2021A Theory of Neural Tangent Kernel Alignment and Its Influence on Trainingcited by 25
  10. 2022What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?cited by 21
  11. 2022Few-shot Backdoor Attacks via Neural Tangent Kernelscited by 15
  12. 2022Evolution of Neural Tangent Kernels under Benign and Adversarial Trainingcited by 14
  13. 2020Overfitting or Underfitting? Understand Robustness Drop in Adversarial Trainingcited by 5

Built on by

Not cited yet.