mlsanityData to 7 Oct 2026

Defense-Prefix for Preventing Typographic Attacks on CLIP

arXiv 2304.04512 ↗First posted 10 Apr 2023cs.CVcited by 24

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language pre-training models (VLPs) have exhibited revolutionary improvements in various vision-language tasks. In VLP, some adversarial attacks fool a model into false or absurd classifications. Previous studies addressed these attacks by fine-tuning the model or changing its architecture. However, these methods risk losing the original model's performance and are difficult to apply to downstream tasks. In particular, their applicability to other tasks has not been considered. In this study, we addressed the reduction of the impact of typographic attacks on CLIP without changing the model parameters. To achieve this, we expand the idea of "prefix learning" and introduce our simple yet effective method: Defense-Prefix (DP), which inserts the DP token before a class name to make words "robust" against typographic attacks. Our method can be easily applied to downstream tasks, such as object detection, because the proposed method is independent of the model parameters. Our method significantly improves the accuracy of classification tasks for typographic attack datasets, while maintaining the zero-shot capabilities of the model. In addition, we leverage our proposed method for object detection, demonstrating its high applicability and effectiveness. The codes and datasets are available at https://github.com/azuma164/Defense-Prefix.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 54 references that are in the map. First posted on arXiv 10 Apr 2023.

Builds on

41 of 54 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  12. 2013Describing Textures in the Wildcited by 2,110
29 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  4. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  5. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  6. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  7. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  8. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  9. 2019How Can We Know What Language Models Know?cited by 953
  10. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  11. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  12. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  13. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  14. 2021Extract Free Dense Labels from CLIPcited by 580
  15. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  16. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  17. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  18. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  19. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  20. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  21. 2022Patching open-vocabulary models by interpolating weightscited by 187
  22. 2022PromptDet: Towards Open-vocabulary Detection using Uncurated Imagescited by 133
  23. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 120
  24. 2022DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotationscited by 102
  25. 2022Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillationcited by 67
  26. 2022CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classificationcited by 65
  27. 2022Disentangling visual and written concepts in CLIPcited by 57
  28. 2022Prefix Conditioning Unifies Language and Label Supervisioncited by 13
  29. 2023Discriminative Class Tokens for Text-to-Image Diffusion Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 6; 2026: 7.

24 papers.

Sort:

By area:

  1. 2025Vision Transformers Don't Need Trained Registerscited by 39
  2. 2024Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attackscited by 36
  3. 2025Steering CLIP's vision transformer with sparse autoencoderscited by 33
  4. 2024Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Modelcited by 22
  5. 2024SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environmentscited by 22
  6. 2024Towards Transferable Attacks Against Vision-LLMs in Autonomous Driving with Typographycited by 17
  7. 2024Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injectioncited by 17
  8. 2024Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Modelscited by 16
  9. 2025SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Modelscited by 12
  10. 2024Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Modelscited by 7
  11. 2024Defending LVLMs Against Vision Attacks through Partial-Perception Supervisioncited by 7
  12. 2024Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Modelscited by 5
12 more
  1. 2025Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Modelscited by 5
  2. 2024Aligning Visual Contrastive learning models via Preference Optimizationcited by 4
  3. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  4. 2025Web Artifact Attacks Disrupt Vision Language Modelscited by 2
  5. 2025Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Modelscited by 2
  6. 2026Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Modelscited by 1
  7. 2026Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Taskscited by 0
  8. 2026One Perturbation, Two Failure Modes: Probing VLM Safety via Embedding-Guided Typographic Perturbationscited by 0
  9. 2026Not What You Asked For: Typographic Attacks in Household Robot Manipulationcited by 0
  10. 2026Towards Robustness against Typographic Attack with Training-free Concept Localizationcited by 0
  11. 2026Beyond Small Patches: Black-Box Detection and Purification of Diverse Backdoor Triggerscited by 0
  12. 2026Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIPcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIP
  2. 2026Typographic Attack Against VLM-based AI-generated Image Detection
  3. 2026QuISE: Defense against Typographic Attacks on VLMs via Query-Irrelevant Semantic Editing
  4. 2023Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks
  5. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective
  6. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
  7. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks
  8. 2022Towards Adversarial Attack on Vision-Language Pre-training Models