mlsanityData to 7 Oct 2026

Universal and Transferable Adversarial Attacks on Aligned Language Models

arXiv 2307.15043 ↗First posted 27 Jul 2023cs.CL cs.AI cs.CRcited by 2,914

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Because "out-of-the-box" large language models are capable of generating a great deal of objectionable content, recent work has focused on aligning these models in an attempt to prevent undesirable generation. While there has been some success at circumventing these measures -- so-called "jailbreaks" against LLMs -- these attacks have required significant human ingenuity and are brittle in practice. In this paper, we propose a simple and effective attack method that causes aligned language models to generate objectionable behaviors. Specifically, our approach finds a suffix that, when attached to a wide range of queries for an LLM to produce objectionable content, aims to maximize the probability that the model produces an affirmative response (rather than refusing to answer). However, instead of relying on manual engineering, our approach automatically produces these adversarial suffixes by a combination of greedy and gradient-based search techniques, and also improves over past automatic prompt generation methods. Surprisingly, we find that the adversarial prompts generated by our approach are quite transferable, including to black-box, publicly released LLMs. Specifically, we train an adversarial attack suffix on multiple prompts (i.e., queries asking for many different types of objectionable content), as well as multiple models (in our case, Vicuna-7B and 13B). When doing so, the resulting attack suffix is able to induce objectionable content in the public interfaces to ChatGPT, Bard, and Claude, as well as open source LLMs such as LLaMA-2-Chat, Pythia, Falcon, and others. In total, this work significantly advances the state-of-the-art in adversarial attacks against aligned language models, raising important questions about how such systems can be prevented from producing objectionable information. Code is available at github.com/llm-attacks/llm-attacks.

Key facts. Cited by 2,914 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 49 references that are in the map. First posted on arXiv 27 Jul 2023.

Builds on

42 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 41

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
29 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  3. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  4. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  5. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  6. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  7. 2016Universal adversarial perturbationscited by 774
  8. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  9. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  10. 2020Aligning AI With Shared Human Valuescited by 706
  11. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  12. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  13. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  14. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  15. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 439
  16. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  17. 2018Generating Natural Language Adversarial Examplescited by 422
  18. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  19. 2023Pretraining Language Models with Human Preferencescited by 245
  20. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  21. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  22. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  23. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  24. 2017The Space of Transferable Adversarial Examplescited by 156
  25. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  26. 2021Globally-Robust Neural Networkscited by 106
  27. 2020SoK: Certified Robustness for Deep Neural Networkscited by 100
  28. 2019Universal Adversarial Perturbations for Speech Recognition Systemscited by 41
  29. 2021Exploring Targeted Universal Adversarial Perturbations to End-to-end ASR Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 155; 2024: 730; 2025: 998; 2026: 1031.

2,914 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  5. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  6. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  7. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  8. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  9. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  10. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  12. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
38 more
  1. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  2. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  3. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  4. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  5. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  6. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  7. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  8. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  9. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  10. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  11. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  12. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  13. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  14. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  15. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  16. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  17. 2023Detecting Language Model Attacks with Perplexitycited by 411
  18. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  19. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  20. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  21. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  22. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  23. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  24. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  25. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  26. 2024A StrongREJECT for Empty Jailbreakscited by 352
  27. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  28. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  29. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  30. 2024Alignment faking in large language modelscited by 316
  31. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  32. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  33. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  34. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  35. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  36. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  37. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  38. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251

50 most cited of 2,914 All 2,914 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings
  2. 2025Scaling Patterns in Adversarial Alignment: Evidence from Multi-LLM Jailbreak Experiments
  3. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  4. 2025The Tower of Babel Revisited: Multilingual Jailbreak Prompts on Closed-Source Large Language Models
  5. 2025LLM Reinforcement in Context
  6. 2024Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Context
  7. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models
  8. 2025Targeting Alignment: Extracting Safety Classifiers of Aligned LLMs