mlsanityData to 7 Oct 2026

Black Box Adversarial Prompting for Foundation Models

arXiv 2302.04237 ↗First posted 8 Feb 2023cs.LGcited by 62

On the map: LLM safety and security → Vision-language and text-to-image safety

Abstract (arXiv)

Prompting interfaces allow users to quickly adjust the output of generative models in both vision and language. However, small changes and design choices in the prompt can lead to significant differences in the output. In this work, we develop a black-box framework for generating adversarial prompts for unstructured image and text generation. These prompts, which can be standalone or prepended to benign prompts, induce specific behaviors into the generative process, such as generating images of a particular object or generating high perplexity text.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security. Builds on 54 of its 72 references that are in the map. First posted on arXiv 8 Feb 2023.

Builds on

54 of 72 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2013Intriguing properties of neural networkscited by 5,114
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
38 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  2. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  4. 2019How Can We Know What Language Models Know?cited by 953
  5. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  6. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  7. 2018A Tutorial on Bayesian Optimizationcited by 734
  8. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  9. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  10. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  11. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  12. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  13. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  14. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  15. 2017Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Modelscited by 459
  16. 2018Generating Natural Language Adversarial Examplescited by 422
  17. 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 411
  18. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 359
  19. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  20. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  21. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  22. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  23. 2019Fooling automated surveillance cameras: adversarial patches to attack person detectioncited by 268
  24. 2019Simple Black-box Adversarial Attackscited by 249
  25. 2019Adversarial Training and Robustness for Multiple Perturbationscited by 182
  26. 2019Certified Robustness to Adversarial Word Substitutionscited by 164
  27. 2018Physical Adversarial Examples for Object Detectorscited by 163
  28. 2018ShapeShifter: Robust Physical Adversarial Attack on Faster R-CNN Object Detectorcited by 159
  29. 2019Making an Invisibility Cloak: Real World Adversarial Attacks on Object Detectorscited by 155
  30. 2018DPatch: An Adversarial Patch Attack on Object Detectorscited by 146
  31. 2018Prior Convictions: Black-Box Adversarial Attacks with Bandits and Priorscited by 145
  32. 2021High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspacescited by 139
  33. 2020Perceptual Adversarial Robustness: Defense Against Unseen Threat Modelscited by 137
  34. 2018Query-Efficient Hard-label Black-box Attack:An Optimization-based Approachcited by 130
  35. 2019Wasserstein Adversarial Examples via Projected Sinkhorn Iterationscited by 89
  36. 2019Adversarial Robustness Against the Union of Multiple Perturbation Modelscited by 88
  37. 2017Exploring the Landscape of Spatial Robustnesscited by 81
  38. 2020Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTcited by 69

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 23; 2025: 20; 2026: 3.

62 papers.

Sort:

By area:

  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  2. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  3. 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 204
  4. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  5. 2023MMA-Diffusion: MultiModal Attack on Diffusion Modelscited by 184
  6. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  7. 2024Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Modelscited by 164
  8. 2023Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Promptscited by 161
  9. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  10. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  11. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  12. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
38 more
  1. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  2. 2023Effective Prompt Extraction from Language Modelscited by 83
  3. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  4. 2024Safeguarding Large Language Models: A Surveycited by 73
  5. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 46
  6. 2023Can LLMs Follow Simple Rules?cited by 45
  7. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  8. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
  9. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  10. 2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Surveycited by 34
  11. 2024Serial Position Effects of Large Language Modelscited by 30
  12. 2024Security of AI Agentscited by 24
  13. 2024Automatic Jailbreaking of the Text-to-Image Generative AI Systemscited by 22
  14. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  15. 2024PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMscited by 20
  16. 2024Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimizationcited by 16
  17. 2024From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditingcited by 15
  18. 2024Acceptable Use Policies for Foundation Modelscited by 12
  19. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspectivecited by 12
  20. 2023Harnessing LLM to Attack LLM-Guarded Text-to-Image Modelscited by 10
  21. 2024ProTIP: Probabilistic Robustness Verification on Text-to-Image Diffusion Models against Stochastic Perturbationcited by 10
  22. 2024SteerDiff: Steering towards Safe Text-to-Image Diffusion Modelscited by 10
  23. 2024RePD: Defending Jailbreak Attack through a Retrieval-based Prompt Decomposition Processcited by 10
  24. 2025Token-Level Constraint Boundary Search for Jailbreaking Text-to-Image Modelscited by 9
  25. 2025Jailbreaking to Jailbreakcited by 8
  26. 2025PLA: Prompt Learning Attack against Text-to-Image Generative Modelscited by 8
  27. 2024Mitigating Dialogue Hallucination for Large Vision Language Models via Adversarial Instruction Tuningcited by 7
  28. 2025AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Promptscited by 6
  29. 2023Joint Composite Latent Space Bayesian Optimizationcited by 5
  30. 2025Dynamic Optimization and Safety Indicator Injection for Jailbreaking Text-to-Image Models with Multimodal Safety Filterscited by 5
  31. 2023A Bayesian approach for prompt optimization in pre-trained language modelscited by 4
  32. 2024Adversarial Attacks on Parts of Speech: An Empirical Study in Text-to-Image Generationcited by 2
  33. 2025The Linear Geometry of Interpretable Tokens: Jailbreaking Attacks and Defenses for Unlearned Diffusion Modelscited by 2
  34. 2025InfoFlood: Jailbreaking Large Language Models with Information Overloadcited by 2
  35. 2025Towards Effective Prompt Stealing Attack against Text-to-Image Diffusion Modelscited by 2
  36. 2026JANUS: A Lightweight Framework for Jailbreaking Text-to-Image Models via Distribution Optimizationcited by 2
  37. 2025PD3F: A Pluggable and Dynamic DoS-Defense Framework Against Resource Consumption Attacks Targeting Large Language Modelscited by 1
  38. 2025Patronus: Safeguarding Text-to-Image Models against White-Box Adversariescited by 1

50 most cited of 62 All 62 (JSON)