mlsanityData to 14 Oct 2026

Query-Based Adversarial Prompt Generation

arXiv 2402.12329 ↗First posted 19 Feb 2024cs.CL cs.AI cs.CRcited by 62

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Recent work has shown it is possible to construct adversarial examples that cause an aligned language model to emit harmful strings or perform harmful behavior. Existing attacks work either in the white-box setting (with full access to the model weights), or through transferability: the phenomenon that adversarial examples crafted on one model often remain effective on other models. We improve on prior work with a query-based attack that leverages API access to a remote language model to construct adversarial examples that cause the model to emit harmful strings with (much) higher probability than with transfer-only attacks. We validate our attack on GPT-3.5 and OpenAI's safety classifier; we can cause GPT-3.5 to emit harmful strings that current transfer attacks fail at, and we can evade the safety classifier with nearly 100% probability.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 39 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

30 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  4. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2023Mistral 7Bcited by 3,164
  8. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2017Evasion Attacks against Machine Learning at Test Timecited by 689
  12. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
14 more
  1. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
  2. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  3. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
  4. 2017Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Modelscited by 462
  5. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  6. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 446
  7. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
  8. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  9. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  10. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 172
  11. 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 137
  12. 2019Improving Black-box Adversarial Attacks with a Transfer-based Priorcited by 121
  13. 2023Language Model Inversioncited by 68
  14. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 50

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 23; 2025: 20; 2026: 19.

62 papers.

Sort:

By area:

  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  2. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  3. 2024Prover-Verifier Games improve legibility of LLM outputscited by 85
  4. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 79
  5. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 79
  6. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
  7. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 70
  8. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  9. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 45
  10. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
  11. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 44
  12. 2024Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characterscited by 42
38 more
  1. 2024Extracting Prompts by Inverting LLM Outputscited by 41
  2. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 31
  3. 2024Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreakingcited by 30
  4. 2025Adversarial Reasoning at Jailbreaking Timecited by 29
  5. 2024Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detectioncited by 23
  6. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
  7. 2025Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interfacecited by 17
  8. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 15
  9. 2024AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lenscited by 14
  10. 2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 14
  11. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 12
  12. 2025Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Modelscited by 10
  13. 2025Universal Jailbreak Suffixes Are Strong Attention Hijackerscited by 10
  14. 2024FLRT: Fluent Student-Teacher Redteamingcited by 9
  15. 2025Smoothed Embeddings for Robust Language Modelscited by 8
  16. 2025Are Robust LLM Fingerprints Adversarially Robust?cited by 8
  17. 2025Black-box Optimization of LLM Outputs by Asking for Directionscited by 7
  18. 2024Are You Human? An Adversarial Benchmark to Expose LLMscited by 5
  19. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 5
  20. 2024LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Secondscited by 4
  21. 2025Toward Principled LLM Safety Testing: Solving the Jailbreak Oracle Problemcited by 4
  22. 2024Towards Universal and Black-Box Query-Response Only Attack on LLMs with QROAcited by 3
  23. 2026Jailbreaking LLMs & VLMs: Mechanisms, Evaluation, and Unified Defensecited by 3
  24. 2026Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Awaycited by 3
  25. 2026Stop Tracking Me! Proactive Defense Against Attribute Inference Attack in LLMscited by 3
  26. 2024On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMscited by 2
  27. 2025Checkpoint-GCG: Auditing and Attacking Fine-Tuning-Based Prompt Injection Defensescited by 2
  28. 2025From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Modelscited by 2
  29. 2025Q-MLLM: Vector Quantization for Robust Multimodal Large Language Model Securitycited by 2
  30. 2025SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Modelscited by 1
  31. 2025Red Teaming Multimodal Language Models: Evaluating Harm Across Prompt Modalities and Modelscited by 1
  32. 2026Accelerating Suffix Jailbreak attacks with Prefix-Shared KV-cachecited by 1
  33. 2026Understanding and Improving Continuous Adversarial Training for LLMs via In-context Learning Theorycited by 1
  34. 2026Stable-GFlowNet: Toward Diverse and Robust LLM Red-Teaming via Contrastive Trajectory Balancecited by 1
  35. 2026A Theoretical Game of Attacks via Compositional Skillscited by 1
  36. 2026EvoJail: Evolutionary Diverse Jailbreak Prompt Generation for Large Language Modelscited by 1
  37. 2026Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directionscited by 1
  38. 2026FinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teamingcited by 1

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Improved Generation of Adversarial Examples Against Safety-aligned LLMs
  2. 2023Certifying LLM Safety against Adversarial Prompting
  3. 2025OET: Optimization-based prompt injection Evaluation Toolkit
  4. 2025Safety Alignment Can Be Not Superficial With Explicit Safety Signals
  5. 2025Safeguarding Efficacy in Large Language Models: Evaluating Resistance to Human-Written and Algorithmic Adversarial Prompts
  6. 2025Early Approaches to Adversarial Fine-Tuning for Prompt Injection Defense: A 2022 Study of GPT-3 and Contemporary Models
  7. 2026RECAP: A Resource-Efficient Method for Adversarial Prompting in Large Language Models
  8. 2023Robust Safety Classifier for Large Language Models: Adversarial Prompt Shield