Robustness Tokens: Towards Adversarial Robustness of Transformers
Not on the map. Why?
Abstract (arXiv)
Recently, large pre-trained foundation models have become widely adopted by machine learning practitioners for a multitude of tasks. Given that such models are publicly available, relying on their use as backbone models for downstream tasks might result in high vulnerability to adversarial attacks crafted with the same public model. In this work, we propose Robustness Tokens, a novel approach specific to the transformer architecture that fine-tunes a few additional private tokens with low computational requirements instead of tuning model parameters as done in traditional adversarial training. We show that Robustness Tokens make Vision Transformer models significantly more robust to white-box adversarial attacks while also retaining the original downstream performances.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 57 of its 64 references that are in the map. First posted on arXiv 13 Mar 2025.
Builds on
57 of 64 references.
References · 57
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
38 more
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2021ViViT: A Video Vision Transformercited by 1,620
- 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
- 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
- 2021Grounded Language-Image Pre-trainingcited by 1,156
- 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
- 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
- 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
- 2023Vision Transformers Need Registerscited by 871
- 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
- 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
- 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
- 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
- 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
- 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
- 2022DeiT III: Revenge of the ViTcited by 419
- 2021Explaining Neural Scaling Lawscited by 363
- 2022Scaling Language-Image Pre-training via Maskingcited by 317
- 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
- 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
- 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
- 2024Massive Activations in Large Language Modelscited by 294
- 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 279
- 2023A Cookbook of Self-Supervised Learningcited by 250
- 2020Torchattacks: A PyTorch Repository for Adversarial Attackscited by 158
- 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
- 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
- 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
- 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
- 2024Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Modelscited by 18
50 most cited of 57 All 57 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.