mlsanityData to 7 Oct 2026

Consistency Regularization for Adversarial Robustness

arXiv 2103.04623 ↗First posted 8 Mar 2021cs.LG cs.AIcited by 42

Not on the map. Why?

Abstract (arXiv)

Adversarial training (AT) is currently one of the most successful methods to obtain the adversarial robustness of deep neural networks. However, the phenomenon of robust overfitting, i.e., the robustness starts to decrease significantly during AT, has been problematic, not only making practitioners consider a bag of tricks for a successful training, e.g., early stopping, but also incurring a significant generalization gap in the robustness. In this paper, we propose an effective regularization technique that prevents robust overfitting by optimizing an auxiliary `consistency' regularization loss during AT. Specifically, we discover that data augmentation is a quite effective tool to mitigate the overfitting in AT, and develop a regularization that forces the predictive distributions after attacking from two different augmentations of the same instance to be similar with each other. Our experimental results demonstrate that such a simple regularization technique brings significant improvements in the test robust accuracy of a wide range of AT methods. More remarkably, we also show that our method could significantly help the model to generalize its robustness against unseen adversaries, e.g., other types or larger perturbations compared to those used during training. Code is available at https://github.com/alinlab/consistency-adversarial.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 47 of its 59 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

47 of 59 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2017On Calibration of Modern Neural Networkscited by 4,860
  8. 2016Wide Residual Networkscited by 3,057
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  10. 2015Fast R-CNNcited by 2,930
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  12. 2020Supervised Contrastive Learningcited by 2,803
35 more
  1. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  2. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  3. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  4. 2016Adversarial examples in the physical worldcited by 1,762
  5. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  6. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  7. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  8. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  9. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  10. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  11. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  12. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  13. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  14. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  15. 2018Robustness May Be at Odds with Accuracycited by 873
  16. 2017Countering Adversarial Images using Input Transformationscited by 517
  17. 2020Overfitting in adversarially robust deep learningcited by 504
  18. 2020Reinforcement Learning with Augmented Datacited by 452
  19. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  20. 2017Defense against Adversarial Attacks Using High-Level Representation Guided Denoisercited by 329
  21. 2018Adversarially Robust Generalization Requires More Datacited by 318
  22. 2018Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionalitycited by 280
  23. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 275
  24. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
  25. 2018Adversarial Logit Pairingcited by 224
  26. 2021Fixing Data Augmentation to Improve Adversarial Robustnesscited by 201
  27. 2020Bag of Tricks for Adversarial Trainingcited by 199
  28. 2019Adversarial Training and Robustness for Multiple Perturbationscited by 182
  29. 2019Adversarial Robustness through Local Linearizationcited by 146
  30. 2020Adversarial Self-Supervised Contrastive Learningcited by 138
  31. 2019Consistency Regularization for Generative Adversarial Networkscited by 134
  32. 2019Adversarial Robustness Against the Union of Multiple Perturbation Modelscited by 88
  33. 2020Consistency Regularization for Certified Robustness of Smoothed Classifierscited by 73
  34. 2019Adversarial camera stickers: A physical camera-based attack on deep learning systemscited by 69
  35. 2019Provable robustness against all adversarial lp-perturbations for p≥ 1cited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 8; 2023: 12; 2024: 8; 2025: 8; 2026: 1.

42 papers.

Sort:

By area:

  1. 2021Unsolved Problems in ML Safetycited by 291
  2. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 207
  3. 2021Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labelscited by 74
  4. 2023Data Augmentation Alone Can Improve Adversarial Trainingcited by 43
  5. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  6. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  7. 2022Understanding and Combating Robust Overfitting via Input Loss Landscape Analysis and Regularizationcited by 22
  8. 2024Adversarial Training on Purification (AToP): Advancing Both Robustness and Generalizationcited by 15
  9. 2022Can Adversarial Training Be Manipulated By Non-Robust Features?cited by 12
  10. 2023Distilling Out-of-Distribution Robustness from Vision-Language Foundation Modelscited by 9
  11. 2024Reliable Model Watermarking: Defending Against Theft without Compromising on Evasioncited by 9
  12. 2021Robustness through Data Augmentation Loss Consistencycited by 8
30 more
  1. 2022A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunitiescited by 8
  2. 2024Machine Learning Robustness: A Primercited by 8
  3. 2023Adversarially Robust Spiking Neural Networks Through Conversioncited by 7
  4. 2021Push Stricter to Decide Better: A Class-Conditional Feature Adaptive Framework for Improving Adversarial Robustnesscited by 6
  5. 2023AROID: Improving Adversarial Robustness Through Online Instance-Wise Data Augmentationcited by 6
  6. 2023Indirect Gradient Matching for Adversarial Robust Distillationcited by 5
  7. 2025Long-tailed Adversarial Training with Self-Distillationcited by 5
  8. 2022Alleviating Robust Overfitting of Adversarial Training With Consistency Regularizationcited by 4
  9. 2025Hierarchical Testing with Rabbit Optimization for Industrial Cyber-Physical Systemscited by 4
  10. 2022Toward Robust Diagnosis: A Contour Attention Preserving Adversarial Defense for COVID-19 Detectioncited by 3
  11. 2023Generating Less Certain Adversarial Examples Improves Robust Generalizationcited by 3
  12. 2024Mitigating Low-Frequency Bias: Feature Recalibration and Frequency Attention Regularization for Adversarial Robustnesscited by 3
  13. 2022Saliency Guided Adversarial Training for Learning Generalizable Features with Applications to Medical Imaging Classification Systemcited by 2
  14. 2024SA-MixNet: Structure-aware Mixup and Invariance Learning for Scribble-supervised Road Extraction in Remote Sensing Imagescited by 2
  15. 2024Improving Adversarial Robustness via Decoupled Visual Representation Maskingcited by 2
  16. 2024New Paradigm of Adversarial Training: Releasing Accuracy-Robustness Trade-Off via Dummy Classcited by 2
  17. 2025Failure Cases Are Better Learned But Boundary Says Sorry: Facilitating Smooth Perception Change for Accuracy-Robustness Trade-Off in Adversarial Trainingcited by 2
  18. 2022Adversarial amplitude swap towards robust image classifierscited by 1
  19. 2023Understanding Robust Overfitting from the Feature Generalization Perspectivecited by 1
  20. 2023ScAR: Scaling Adversarial Robustness for LiDAR Object Detectioncited by 1
  21. 2024MEAT: Median-Ensemble Adversarial Training for Improving Robustness and Generalizationcited by 1
  22. 2025Boosting Adversarial Robustness and Generalization with Dictionary Structurecited by 1
  23. 2021Comparing the Machine Readability of Traffic Sign Pictograms in Austria and Germanycited by 0
  24. 2023Exploiting Frequency Spectrum of Adversarial Images for General Robustnesscited by 0
  25. 2023A Spectral Perspective towards Understanding and Improving Adversarial Robustnesscited by 0
  26. 2023Improving Robustness via Tilted Exponential Layer: A Communication-Theoretic Perspectivecited by 0
  27. 2025Adversarially Robust Spiking Neural Networks with Sparse Connectivitycited by 0
  28. 2025Towards Trustworthy Wi-Fi CSI-based Sensing: Systematic Evaluation of Adversarial Robustnesscited by 0
  29. 2025Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillationcited by 0
  30. 2026Robust Alignment: Harmonizing Clean Accuracy and Adversarial Robustness in Adversarial Trainingcited by 0