mlsanityData to 7 Oct 2026

Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation

arXiv 2311.03348 ↗First posted 6 Nov 2023cs.CL cs.AI cs.LGcited by 202

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Despite efforts to align large language models to produce harmless responses, they are still vulnerable to jailbreak prompts that elicit unrestricted behaviour. In this work, we investigate persona modulation as a black-box jailbreaking method to steer a target model to take on personalities that are willing to comply with harmful instructions. Rather than manually crafting prompts for each persona, we automate the generation of jailbreaks using a language model assistant. We demonstrate a range of harmful completions made possible by persona modulation, including detailed instructions for synthesising methamphetamine, building a bomb, and laundering money. These automated attacks achieve a harmful completion rate of 42.5% in GPT-4, which is 185 times larger than before modulation (0.23%). These prompts also transfer to Claude 2 and Vicuna with harmful completion rates of 61.0% and 35.9%, respectively. Our work reveals yet another vulnerability in commercial large language models and highlights the need for more comprehensive safeguards.

Key facts. Cited by 202 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 20 of its 30 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

20 of 30 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  2. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  4. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  5. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  9. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  10. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  11. 2023Are aligned neural networks adversarially aligned?cited by 305
  12. 2023Role-Play with Large Language Modelscited by 290
3 more
  1. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  2. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  3. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 81; 2025: 73; 2026: 44.

202 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  3. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  5. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  6. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  7. 2024A StrongREJECT for Empty Jailbreakscited by 352
  8. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  9. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  10. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  11. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  12. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
38 more
  1. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  2. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  3. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  4. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 132
  5. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  6. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  7. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  8. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  9. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  10. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
  11. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  12. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  13. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  14. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  15. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  16. 2025Persona Features Control Emergent Misalignmentcited by 76
  17. 2024Best-of-N Jailbreakingcited by 71
  18. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 70
  19. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothingcited by 68
  20. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  21. 2025Safety in Large Reasoning Models: A Surveycited by 58
  22. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
  23. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  24. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 46
  25. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 46
  26. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 45
  27. 2024GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Modelscited by 44
  28. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 42
  29. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 41
  30. 2024Voice Jailbreak Attacks Against GPT-4ocited by 39
  31. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  32. 2025AgentSafe: Safeguarding Large Language Model-based Multi-agent Systems via Hierarchical Data Managementcited by 35
  33. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33
  34. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  35. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 31
  36. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulationcited by 31
  37. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  38. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28

50 most cited of 202 All 202 (JSON)