mlsanityData to 14 Oct 2026

Pyramid Adversarial Training Improves ViT Performance

arXiv 2111.15121 ↗First posted 30 Nov 2021cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

Aggressive data augmentation is a key component of the strong generalization capabilities of Vision Transformer (ViT). One such data augmentation technique is adversarial training (AT); however, many prior works have shown that this often results in poor clean accuracy. In this work, we present pyramid adversarial training (PyramidAT), a simple and effective technique to improve ViT's overall performance. We pair it with a "matched" Dropout and stochastic depth regularization, which adopts the same Dropout and stochastic depth configuration for the clean and adversarial samples. Similar to the improvements on CNNs by AdvProp (not directly applicable to ViT), our pyramid adversarial training breaks the trade-off between in-distribution accuracy and out-of-distribution robustness for ViT and related architectures. It leads to 1.82% absolute improvement on ImageNet clean accuracy for the ViT-B model when trained only on ImageNet-1K data, while simultaneously boosting performance on 7 ImageNet robustness metrics, by absolute numbers ranging from 1.76% to 15.68%. We set a new state-of-the-art for ImageNet-C (41.42 mCE), ImageNet-R (53.92%), and ImageNet-Sketch (41.04%) without extra data, using only the ViT-B/16 backbone and our pyramid adversarial training. Our code is publicly available at pyramidat.github.io.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 56 of its 67 references that are in the map. First posted on arXiv 30 Nov 2021.

Builds on

56 of 67 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2013Intriguing properties of neural networkscited by 5,155
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  2. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  8. 2021Vision Transformers for Dense Predictioncited by 1,803
  9. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  10. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  12. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  13. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  14. 2019Natural Adversarial Examplescited by 1,408
  15. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  16. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  17. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  18. 2016Deep Networks with Stochastic Depthcited by 979
  19. 2016Adversarial Machine Learning at Scalecited by 936
  20. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  21. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  22. 2018Robustness May Be at Odds with Accuracycited by 880
  23. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
  24. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  25. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  26. 2021Intriguing Properties of Vision Transformerscited by 446
  27. 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
  28. 2020Are we done with ImageNet?cited by 333
  29. 2019A Fourier Perspective on Model Robustness in Computer Visioncited by 288
  30. 2019Adversarial Examples Improve Image Recognitioncited by 286
  31. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  32. 2018Spatially Transformed Adversarial Examplescited by 195
  33. 2018Robustness via curvature regularization, and vice versacited by 175
  34. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  35. 2020Understanding and Mitigating the Tradeoff Between Robustness and Accuracycited by 149
  36. 2019Adversarial Robustness through Local Linearizationcited by 146
  37. 2021Towards Robust Vision Transformercited by 137
  38. 2019Robust Neural Machine Translation with Doubly Adversarial Inputscited by 106

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 16; 2023: 8; 2024: 12; 2025: 2; 2026: 1.

39 papers.

Sort:

By area:

  1. 2022Improving Vision Transformers by Revisiting High-frequency Componentscited by 76
  2. 2022LPT: Long-tailed Prompt Tuning for Image Classificationcited by 69
  3. 2023How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defensescited by 66
  4. 2022A Light Recipe to Train Robust Vision Transformerscited by 64
  5. 2022Context-Aware Robust Fine-Tuningcited by 28
  6. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representationscited by 25
  7. 2023Zero-Shot Learning by Harnessing Adversarial Samplescited by 24
  8. 2022Enhance the Visual Representation via Discrete Adversarial Trainingcited by 23
  9. 2023Seasoning Model Soups for Robustness to Adversarial and Natural Distribution Shiftscited by 23
  10. 2022A Comprehensive Study of the Robustness for LiDAR-based 3D Object Detectors against Adversarial Attackscited by 22
  11. 2022Revisiting adapters with adversarial trainingcited by 18
  12. 2024Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Modelscited by 18
27 more
  1. 2022Improving Out-of-Distribution Generalization by Adversarial Training with Structured Priorscited by 17
  2. 2024HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generationcited by 14
  3. 2022Infrared Image Super-Resolution: Systematic Review, and Future Trendscited by 11
  4. 2023Distilling Out-of-Distribution Robustness from Vision-Language Foundation Modelscited by 9
  5. 2024Intriguing Equivalence Structures of the Embedding Space of Vision Transformerscited by 8
  6. 2022Revisiting the Adversarial Robustness-Accuracy Tradeoff in Robot Learningcited by 7
  7. 2022Fast AdvPropcited by 6
  8. 2024Adversarial Training: A Surveycited by 6
  9. 2022Deeper Insights into the Robustness of ViTs towards Common Corruptionscited by 5
  10. 2022ColorSense: A Study on Color Vision in Machine Visual Recognitioncited by 5
  11. 2023Steerable Equivariant Representation Learningcited by 5
  12. 2024Few-Shot Unsupervised Implicit Neural Shape Representation Learning with Spatial Adversariescited by 5
  13. 2024Fully Attentional Networks with Self-emerging Token Labelingcited by 3
  14. 2025Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generationcited by 3
  15. 2024Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinementcited by 2
  16. 2024Machine unlearning through fine-grained model parameters perturbationcited by 2
  17. 2024Intriguing Differences Between Zero-Shot and Systematic Evaluations of Vision-Language Transformer Modelscited by 2
  18. 2022Robustness and invariance properties of image classifierscited by 1
  19. 2023Blacksmith: Fast Adversarial Training of Vision Transformers via a Mixture of Single-step and Multi-step Methodscited by 1
  20. 2024Clarifying Myths About the Relationship Between Shape Bias, Accuracy, and Robustnesscited by 1
  21. 2024LPT++: Efficient Training on Mixture of Long-tailed Expertscited by 1
  22. 2022MixBoost: Improving the Robustness of Deep Neural Networks by Boosting Data Augmentationcited by 0
  23. 2023Infrared Image Super-Resolution via GANcited by 0
  24. 2023Universal Pyramid Adversarial Training for Improved ViT Performancecited by 0
  25. 2024EntProp: High Entropy Propagation for Improving Accuracy and Robustnesscited by 0
  26. 2025EasyRobust: A Comprehensive and Easy-to-use Toolkit for Robust and Generalized Visioncited by 0
  27. 2026Investigating Adversarial Robustness of Multi-modal Large Language Modelscited by 0