# Vision Transformers: the threat of realistic adversarial patches

arXiv: https://arxiv.org/abs/2509.21084 · HTML: https://mlsanity.com/paper/2509.21084/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Adversarial Patch](https://mlsanity.com/paper/1712.09665/) (2017)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Fooling automated surveillance cameras: adversarial patches to attack person detection](https://mlsanity.com/paper/1904.08653/) (2019)
- [On the Robustness of Vision Transformers to Adversarial Examples](https://mlsanity.com/paper/2104.02610/) (2021)
- [Making an Invisibility Cloak: Real World Adversarial Attacks on Object Detectors](https://mlsanity.com/paper/1910.14667/) (2019)
- [AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learning](https://mlsanity.com/paper/2308.07026/) (2023)
- [Towards Transferable Adversarial Attacks on Vision Transformers](https://mlsanity.com/paper/2109.04176/) (2021)
- [Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization](https://mlsanity.com/paper/2303.15754/) (2023)
- [DAP: A Dynamic Adversarial Patch for Evading Person Detectors](https://mlsanity.com/paper/2305.11618/) (2023)
- [Are Vision Transformers Robust to Patch Perturbations?](https://mlsanity.com/paper/2111.10659/) (2021)
- [Contrastive Language-Image Pre-Training with Knowledge Graphs](https://mlsanity.com/paper/2210.08901/) (2022)
- [Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey](https://mlsanity.com/paper/2410.23687/) (2024)
- [Transformer for Object Re-Identification: A Survey](https://mlsanity.com/paper/2401.06960/) (2024)
- [Physical Adversarial Attacks for Surveillance: A Survey](https://mlsanity.com/paper/2305.01074/) (2023)
- [Generalized Single-Image-Based Morphing Attack Detection Using Deep Representations from Vision Transformer](https://mlsanity.com/paper/2501.09817/) (2025)
- [Backdoor Attacks on Vision Transformers](https://mlsanity.com/paper/2206.08477/) (2022)
- [Toward a Holistic Evaluation of Robustness in CLIP Models](https://mlsanity.com/paper/2410.01534/) (2024)
- [CLIP-Decoder : ZeroShot Multilabel Classification using Multimodal CLIP Aligned Representation](https://mlsanity.com/paper/2406.14830/) (2024)
- [Mechanistic Understandings of Representation Vulnerabilities and Engineering Robust Vision Transformers](https://mlsanity.com/paper/2502.04679/) (2025)
- [Random Position Adversarial Patch for Vision Transformers](https://mlsanity.com/paper/2307.04066/) (2023)

## Abstract (arXiv)

The increasing reliance on machine learning systems has made their security a critical concern. Evasion attacks enable adversaries to manipulate the decision-making processes of AI systems, potentially causing security breaches or misclassification of targets. Vision Transformers (ViTs) have gained significant traction in modern machine learning due to increased 1) performance compared to Convolutional Neural Networks (CNNs) and 2) robustness against adversarial perturbations. However, ViTs remain vulnerable to evasion attacks, particularly to adversarial patches, unique patterns designed to manipulate AI classification systems. These vulnerabilities are investigated by designing realistic adversarial patches to cause misclassification in person vs. non-person classification tasks using the Creases Transformation (CT) technique, which adds subtle geometric distortions similar to those occurring naturally when wearing clothing. This study investigates the transferability of adversarial attack techniques used in CNNs when applied to ViT classification models. Experimental evaluation across four fine-tuned ViT models on a binary person classification task reveals significant vulnerability variations: attack success rates ranged from 40.04% (google/vit-base-patch16-224-in21k) to 99.97% (facebook/dino-vitb16), with google/vit-base-patch16-224 achieving 66.40% and facebook/dinov3-vitb16 reaching 65.17%. These results confirm the cross-architectural transferability of adversarial patches from CNNs to ViTs, with pre-training dataset scale and methodology strongly influencing model resilience to adversarial attacks.