# Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization

arXiv: https://arxiv.org/abs/2303.15754 · HTML: https://mlsanity.com/paper/2303.15754/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018)
- [ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases](https://mlsanity.com/paper/2103.10697/) (2021)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks](https://mlsanity.com/paper/1904.02884/) (2019)
- [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021)
- [Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks](https://mlsanity.com/paper/1908.06281/) (2019)
- [Understanding Robustness of Transformers for Image Classification](https://mlsanity.com/paper/2103.14586/) (2021)
- [Enhancing the Transferability of Adversarial Attacks through Variance Tuning](https://mlsanity.com/paper/2103.15571/) (2021)
- [Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets](https://mlsanity.com/paper/2002.05990/) (2020)
- [On the Robustness of Vision Transformers to Adversarial Examples](https://mlsanity.com/paper/2104.02610/) (2021)
- [Visformer: The Vision-friendly Transformer](https://mlsanity.com/paper/2104.12533/) (2021)
- [On the Adversarial Robustness of Vision Transformers](https://mlsanity.com/paper/2103.15670/) (2021)
- [Improving Adversarial Transferability via Neuron Attribution-Based Attacks](https://mlsanity.com/paper/2204.00008/) (2022)
- [Towards Transferable Adversarial Attacks on Vision Transformers](https://mlsanity.com/paper/2109.04176/) (2021)
- [On Improving Adversarial Transferability of Vision Transformers](https://mlsanity.com/paper/2106.04169/) (2021)
- [Boosting Adversarial Transferability through Enhanced Momentum](https://mlsanity.com/paper/2103.10609/) (2021)
- [Triangle Attack: A Query-efficient Decision-based Adversarial Attack](https://mlsanity.com/paper/2112.06569/) (2021)

## Built on by (most cited first, 30 of 42)

- [How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses](https://mlsanity.com/paper/2305.10862/) (2023)
- [SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation](https://mlsanity.com/paper/2312.04913/) (2023)
- [On the Robustness of Latent Diffusion Models](https://mlsanity.com/paper/2306.08257/) (2023)
- [Bag of Tricks to Boost Adversarial Transferability](https://mlsanity.com/paper/2401.08734/) (2024)
- [As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?](https://mlsanity.com/paper/2403.12693/) (2024)
- [Large Language Models are Good Multi-lingual Learners : When LLMs Meet Cross-lingual Prompts](https://mlsanity.com/paper/2409.11056/) (2024)
- [Towards Model Resistant to Transferable Adversarial Examples via Trigger Activation](https://mlsanity.com/paper/2504.14541/) (2025)
- [AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization](https://mlsanity.com/paper/2504.01735/) (2025)
- [Improving Adversarial Transferability on Vision Transformers via Forward Propagation Refinement](https://mlsanity.com/paper/2503.15404/) (2025)
- [FOCUS: Fused Observation of Channels for Unveiling Spectra](https://mlsanity.com/paper/2507.14787/) (2025)
- [Adversarial Training: A Survey](https://mlsanity.com/paper/2410.15042/) (2024)
- [Benchmarking Transferable Adversarial Attacks](https://mlsanity.com/paper/2402.00418/) (2024)
- [Benchmarking Unified Face Attack Detection via Hierarchical Prompt Tuning](https://mlsanity.com/paper/2505.13327/) (2025)
- [Identifying Performance Issues in Cloud Service Systems Based on Relational-Temporal Features](https://mlsanity.com/paper/2307.10869/) (2023)
- [Backpropagation Path Search On Adversarial Transferability](https://mlsanity.com/paper/2308.07625/) (2023)
- [Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability](https://mlsanity.com/paper/2504.10804/) (2025)
- [Attacking Transformers with Feature Diversity Adversarial Perturbation](https://mlsanity.com/paper/2403.07942/) (2024)
- [Boosting Generative Adversarial Transferability with Self-supervised Vision Transformer Features](https://mlsanity.com/paper/2506.21046/) (2025)
- [Vulnerability Analysis of Transformer-based Optical Character Recognition to Adversarial Attacks](https://mlsanity.com/paper/2311.17128/) (2023)
- [QuantAttack: Exploiting Dynamic Quantization to Attack Vision Transformers](https://mlsanity.com/paper/2312.02220/) (2023)
- [DeSparsify: Adversarial Attack Against Token Sparsification Mechanisms in Vision Transformers](https://mlsanity.com/paper/2402.02554/) (2024)
- [Artificial Immune System of Secure Face Recognition Against Adversarial Attacks](https://mlsanity.com/paper/2406.18144/) (2024)
- [ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformer](https://mlsanity.com/paper/2409.13828/) (2024)
- [Boosting the Local Invariance for Better Adversarial Transferability](https://mlsanity.com/paper/2503.06140/) (2025)
- [TESSER: Transfer-Enhancing Adversarial Attacks from Vision Transformers via Spectral and Semantic Regularization](https://mlsanity.com/paper/2505.19613/) (2025)
- [Attacking Attention of Foundation Models Disrupts Downstream Tasks](https://mlsanity.com/paper/2506.05394/) (2025)
- [Frequency-Domain Regularized Adversarial Alignment for Transferable Attacks against Closed-Source MLLMs](https://mlsanity.com/paper/2605.21541/) (2026)
- [Semantic-Syntactic Discrepancy in Images (SSDI): Learning Meaning and Order of Features from Natural Images](https://mlsanity.com/paper/2401.17515/) (2024)
- [On Inherent Adversarial Robustness of Active Vision Systems](https://mlsanity.com/paper/2404.00185/) (2024)
- [Downstream Transfer Attack: Adversarial Attacks on Downstream Models with Pre-trained Vision Transformers](https://mlsanity.com/paper/2408.01705/) (2024)

## Abstract (arXiv)

Vision transformers (ViTs) have been successfully deployed in a variety of computer vision tasks, but they are still vulnerable to adversarial samples. Transfer-based attacks use a local model to generate adversarial samples and directly transfer them to attack a target black-box model. The high efficiency of transfer-based attacks makes it a severe security threat to ViT-based applications. Therefore, it is vital to design effective transfer-based attacks to identify the deficiencies of ViTs beforehand in security-sensitive scenarios. Existing efforts generally focus on regularizing the input gradients to stabilize the updated direction of adversarial samples. However, the variance of the back-propagated gradients in intermediate blocks of ViTs may still be large, which may make the generated adversarial samples focus on some model-specific features and get stuck in poor local optima. To overcome the shortcomings of existing approaches, we propose the Token Gradient Regularization (TGR) method. According to the structural characteristics of ViTs, TGR reduces the variance of the back-propagated gradient in each internal block of ViTs in a token-wise manner and utilizes the regularized gradient to generate adversarial samples. Extensive experiments on attacking both ViTs and CNNs confirm the superiority of our approach. Notably, compared to the state-of-the-art transfer-based attacks, our TGR offers a performance improvement of 8.8% on average.