mlsanityData to 14 Oct 2026

Towards Robust and Accurate Visual Prompting

arXiv 2311.10992 ↗First posted 18 Nov 2023cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Visual prompting, an efficient method for transfer learning, has shown its potential in vision tasks. However, previous works focus exclusively on VP from standard source models, it is still unknown how it performs under the scenario of a robust source model: Whether a visual prompt derived from a robust model can inherit the robustness while suffering from the generalization performance decline, albeit for a downstream dataset that is different from the source dataset? In this work, we get an affirmative answer of the above question and give an explanation on the visual representation level. Moreover, we introduce a novel technique named Prompt Boundary Loose (PBL) to effectively mitigates the suboptimal results of visual prompt on standard accuracy without losing (or even significantly improving) its adversarial robustness when using a robust model as source model. Extensive experiments across various datasets show that our findings are universal and demonstrate the significant benefits of our proposed method.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 53 of its 68 references that are in the map. First posted on arXiv 18 Nov 2023.

Builds on

53 of 68 references.

References · 53

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  12. 2013Describing Textures in the Wildcited by 2,125
38 more
  1. 2022Visual Prompt Tuningcited by 1,681
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  3. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  4. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  6. 2016Adversarial Machine Learning at Scalecited by 936
  7. 2016Practical Black-Box Attacks against Machine Learningcited by 903
  8. 2018Robustness May Be at Odds with Accuracycited by 880
  9. 2017Ensemble Adversarial Training: Attacks and Defensescited by 840
  10. 2021Robust fine-tuning of zero-shot modelscited by 797
  11. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  12. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
  13. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  14. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 685
  15. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  16. 2019Adversarial Training for Free!cited by 643
  17. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 501
  18. 2017Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Modelscited by 462
  19. 2017Defense against Adversarial Attacks Using High-Level Representation Guided Denoisercited by 329
  20. 2019Adversarial Examples Improve Image Recognitioncited by 286
  21. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 278
  22. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  23. 2018Adversarial Attacks and Defences: A Surveycited by 259
  24. 2019Simple Black-box Adversarial Attackscited by 251
  25. 2020Understanding and Improving Fast Adversarial Trainingcited by 214
  26. 2019Adversarial Training and Robustness for Multiple Perturbationscited by 183
  27. 2019Improving Black-box Adversarial Attacks with a Transfer-based Priorcited by 121
  28. 2018Adversarial Reprogramming of Neural Networkscited by 116
  29. 2022Robustness and Accuracy Could Be Reconcilable by (Proper) Definitioncited by 111
  30. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  31. 2020RayS: A Ray Searching Method for Hard-label Adversarial Attackcited by 96
  32. 2020Smooth Adversarial Trainingcited by 89
  33. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 87
  34. 2020Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resourcescited by 83
  35. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  36. 2018Adversarial Examples: Opportunities and Challengescited by 63
  37. 2018Bilateral Adversarial Training: Towards Fast Training of More Robust Models Against Adversarial Attackscited by 48
  38. 2019Jacobian Adversarially Regularized Networks for Robustnesscited by 40

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  2. 2025GraphBridge: Towards Arbitrary Transfer Learning in GNNscited by 4
  3. 2024Encapsulating Knowledge in One Promptcited by 3
  4. 2026Refinement Provenance Inference: Detecting LLM-Refined Training Prompts from Model Behaviorcited by 3
  5. 2025Multi-Level Collaboration in Model Mergingcited by 0