# Visual Prompting for Adversarial Robustness

arXiv: https://arxiv.org/abs/2210.06284 · HTML: https://mlsanity.com/paper/2210.06284/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 50 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Adversarial Training for Free!](https://mlsanity.com/paper/1904.12843/) (2019)
- [Provable defenses against adversarial examples via the convex outer adversarial polytope](https://mlsanity.com/paper/1711.00851/) (2017)
- [Unlabeled Data Improves Adversarial Robustness](https://mlsanity.com/paper/1905.13736/) (2019)
- [Feature Denoising for Improving Adversarial Robustness](https://mlsanity.com/paper/1812.03411/) (2018)
- [Certified Defenses against Adversarial Examples](https://mlsanity.com/paper/1801.09344/) (2018)
- [MagNet: a Two-Pronged Defense against Adversarial Examples](https://mlsanity.com/paper/1705.09064/) (2017)
- [On Detecting Adversarial Perturbations](https://mlsanity.com/paper/1702.04267/) (2017)
- [You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle](https://mlsanity.com/paper/1905.00877/) (2019)
- [On the (Statistical) Detection of Adversarial Examples](https://mlsanity.com/paper/1702.06280/) (2017)
- [Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning](https://mlsanity.com/paper/2003.12862/) (2020)
- [Adversarial purification with Score-based generative models](https://mlsanity.com/paper/2106.06041/) (2021)
- [Adversarial Reprogramming of Neural Networks](https://mlsanity.com/paper/1806.11146/) (2018)
- [Adversarial Robustness vs Model Compression, or Both?](https://mlsanity.com/paper/1903.12561/) (2019)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [Stable Neural ODE with Lyapunov-Stable Equilibrium Points for Defending Against Adversarial Attacks](https://mlsanity.com/paper/2110.12976/) (2021)
- [Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources](https://mlsanity.com/paper/2007.08714/) (2020)
- [When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?](https://mlsanity.com/paper/2111.01124/) (2021)
- [Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning](https://mlsanity.com/paper/2202.10629/) (2022)
- [Evaluating the Adversarial Robustness of Adaptive Test-time Defenses](https://mlsanity.com/paper/2202.13711/) (2022)
- [ML-LOO: Detecting Adversarial Examples with Feature Attribution](https://mlsanity.com/paper/1906.03499/) (2019)

## Built on by (most cited first, 30 of 34)

- [PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning](https://mlsanity.com/paper/2211.11682/) (2022)
- [Understanding and Improving Visual Prompting: A Label-Mapping Perspective](https://mlsanity.com/paper/2211.11635/) (2022)
- [One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2403.01849/) (2024)
- [AutoVP: An Automated Visual Prompting Framework and Benchmark](https://mlsanity.com/paper/2310.08381/) (2023)
- [Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection](https://mlsanity.com/paper/2406.00510/) (2024)
- [Text-Visual Prompting for Efficient 2D Temporal Video Grounding](https://mlsanity.com/paper/2303.04995/) (2023)
- [Exploring the Benefits of Visual Prompting in Differential Privacy](https://mlsanity.com/paper/2303.12247/) (2023)
- [Improving Adversarial Robustness of Masked Autoencoders via Test-time Frequency-domain Prompting](https://mlsanity.com/paper/2308.10315/) (2023)
- [Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning](https://mlsanity.com/paper/2310.08782/) (2023)
- [Adversarial Examples in the Physical World: A Survey](https://mlsanity.com/paper/2311.01473/) (2023)
- [Subgraph-level Universal Prompt Tuning](https://mlsanity.com/paper/2402.10380/) (2024)
- [Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Models](https://mlsanity.com/paper/2505.15130/) (2025)
- [Quantifying Privacy Risks of Prompts in Visual Prompt Learning](https://mlsanity.com/paper/2310.11970/) (2023)
- [DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Prompt](https://mlsanity.com/paper/2506.09353/) (2025)
- [Adversarial Machine Learning for Social Good: Reframing the Adversary as an Ally](https://mlsanity.com/paper/2310.03614/) (2023)
- [Adversarial Prompt Distillation for Vision-Language Models](https://mlsanity.com/paper/2411.15244/) (2024)
- [Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Models](https://mlsanity.com/paper/2503.12874/) (2025)
- [Towards Robust and Accurate Visual Prompting](https://mlsanity.com/paper/2311.10992/) (2023)
- [Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transformers](https://mlsanity.com/paper/2405.10612/) (2024)
- [Prompt Backdoors in Visual Prompt Learning](https://mlsanity.com/paper/2310.07632/) (2023)
- [Encapsulating Knowledge in One Prompt](https://mlsanity.com/paper/2407.11902/) (2024)
- [Towards Ambiguity-Free Spatial Foundation Model: Rethinking and Decoupling Depth Ambiguity](https://mlsanity.com/paper/2503.06014/) (2025)
- [Adversarial Examples are Misaligned in Diffusion Model Manifolds](https://mlsanity.com/paper/2401.06637/) (2024)
- [Enhancing Parameter-Efficient Fine-Tuning of Vision Transformers through Frequency-Based Adaptation](https://mlsanity.com/paper/2411.19297/) (2024)
- [Invisible Watermarks, Visible Gains: Steering Machine Unlearning with Bi-Level Watermarking Design](https://mlsanity.com/paper/2508.10065/) (2025)
- [FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2509.06992/) (2025)
- [Visual prompting reimagined: The power of the Activation Prompts](https://mlsanity.com/paper/2604.06440/) (2026)
- [ADAPT to Robustify Prompt Tuning Vision Transformers](https://mlsanity.com/paper/2403.13196/) (2024)
- [E2MPL:An Enduring and Efficient Meta Prompt Learning Framework for Few-shot Unsupervised Domain Adaptation](https://mlsanity.com/paper/2407.04066/) (2024)
- [Improving Adversarial Robustness via Phase and Amplitude-aware Prompting](https://mlsanity.com/paper/2502.03758/) (2025)

## Abstract (arXiv)

In this work, we leverage visual prompting (VP) to improve adversarial robustness of a fixed, pre-trained model at testing time. Compared to conventional adversarial defenses, VP allows us to design universal (i.e., data-agnostic) input prompting templates, which have plug-and-play capabilities at testing time to achieve desired model performance without introducing much computation overhead. Although VP has been successfully applied to improving model generalization, it remains elusive whether and how it can be used to defend against adversarial attacks. We investigate this problem and show that the vanilla VP approach is not effective in adversarial defense since a universal input prompt lacks the capacity for robust learning against sample-specific adversarial perturbations. To circumvent it, we propose a new VP method, termed Class-wise Adversarial Visual Prompting (C-AVP), to generate class-wise visual prompts so as to not only leverage the strengths of ensemble prompts but also optimize their interrelations to improve model robustness. Our experiments show that C-AVP outperforms the conventional VP method, with 2.1X standard accuracy gain and 2X robust accuracy gain. Compared to classical test-time defenses, C-AVP also yields a 42X inference time speedup.