mlsanityData to 14 Oct 2026

Approximate Nullspace Augmented Finetuning for Robust Vision Transformers

arXiv 2403.10476 ↗First posted 15 Mar 2024cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Enhancing the robustness of deep learning models, particularly in the realm of vision transformers (ViTs), is crucial for their real-world deployment. In this work, we provide a finetuning approach to enhance the robustness of vision transformers inspired by the concept of nullspace from linear algebra. Our investigation centers on whether a vision transformer can exhibit resilience to input variations akin to the nullspace property in linear mappings, which would imply that perturbations sampled from this nullspace do not influence the model's output when added to the input. We start from the observation that many existing ViTs satisfy this property because their patch embedding layer has a non-trivial nullspace. Then, we extend the notion of nullspace to nonlinear settings and demonstrate that it is possible to synthesize approximate nullspace elements for ViT's encoder blocks through optimization. Finally, we propose a finetuning strategy for ViTs wherein we augment the training data with synthesized approximate nullspace noise. We find that our finetuning approach significantly improves the models' robustness to both adversarial and natural image perturbations.footnoteCode is available at: https://github.com/Liu-Hy/ns-vit.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 59 of its 76 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

59 of 76 references.

References · 59

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2023Segment Anythingcited by 8,234
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
38 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  2. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  3. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  4. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  5. 2019Natural Adversarial Examplescited by 1,408
  6. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
  7. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  8. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
  9. 2020Quantifying Attention Flow in Transformerscited by 775
  10. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 766
  11. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  12. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  13. 2020Overfitting in adversarially robust deep learningcited by 506
  14. 2017Interpretation of Neural Networks is Fragilecited by 444
  15. 2018Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooringcited by 357
  16. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  17. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 278
  18. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  19. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 227
  20. 2019Explanations can be manipulated and geometry is to blamecited by 216
  21. 2021Vision Transformers are Robust Learnerscited by 207
  22. 2021Data Augmentation Can Improve Robustnesscited by 194
  23. 2021Are Transformers More Robust Than CNNs?cited by 187
  24. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  25. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 143
  26. 2017Adversarial Frontier Stitching for Remote Neural Network Watermarkingcited by 118
  27. 2019Fooling Neural Network Interpretations via Adversarial Model Manipulationcited by 112
  28. 2021On the Adversarial Robustness of Vision Transformerscited by 106
  29. 2021Contextual Transformer Networks for Visual Recognitioncited by 90
  30. 2020On the Benefits of Invariance in Neural Networkscited by 79
  31. 2020DADA: Differentiable Automatic Data Augmentationcited by 57
  32. 2023AttentionViz: A Global View of Transformer Attentioncited by 50
  33. 2022Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?cited by 49
  34. 2022TokenMix: Rethinking Image Mixing for Data Augmentation in Vision Transformerscited by 48
  35. 2020Universal Adversarial Attack on Attention and the Resulting Dataset DAmageNetcited by 45
  36. 2022Optimizing Relevance Maps of Vision Transformers Improves Robustnesscited by 41
  37. 2021Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentationcited by 40
  38. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.