mlsanityData to 14 Oct 2026

Jailbreaking to Jailbreak

arXiv 2502.09638 ↗First posted 9 Feb 2025cs.CL cs.AIcited by 9

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs) can be used to red team other models (e.g. jailbreaking) to elicit harmful contents. While prior works commonly employ open-weight models or private uncensored models for doing jailbreaking, as the refusal-training of strong LLMs (e.g. OpenAI o3) refuse to help jailbreaking, our work turn (almost) any black-box LLMs into attackers. The resulting J2 (jailbreaking-to-jailbreak) attackers can effectively jailbreak the safeguard of target models using various strategies, both created by themselves or from expert human red teamers. In doing so, we show their strong but under-researched jailbreaking capabilities. Our experiments demonstrate that 1) prompts used to create J2 attackers transfer across almost all black-box models; 2) an J2 attacker can jailbreak a copy of itself, and this vulnerability develops rapidly over the past 12 months; 3) reasong models, such as Sonnet-3.7, are strong J2 attackers compared to others. For example, when used against the safeguard of GPT-4o, J2 (Sonnet-3.7) achieves 0.975 attack success rate (ASR), which matches expert human red teamers and surpasses the state-of-the-art algorithm-based attacks. Among J2 attackers, J2 (o3) achieves highest ASR (0.605) against Sonnet-3.5, one of the most robust models.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 45 of its 60 references that are in the map. First posted on arXiv 9 Feb 2025.

Builds on

45 of 60 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Adversarial Reasoning at Jailbreaking Time3 Feb
  2. 2024Endless Jailbreaks with Bijection Learning2 Oct
  3. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet27 Aug
  4. 2024Multi-Turn Context Jailbreak Attack on Large Language Models From First Principles8 Aug
  5. 2024Tamper-Resistant Safeguards for Open-Weight LLMs1 Aug
  6. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs22 Jul
  7. 2024Efficient Adversarial Training in LLMs with Continuous Attacks24 May
  8. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  9. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack2 Apr
  10. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
  11. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks30 Jan
  12. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
5 more
  1. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  2. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  4. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  5. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  9. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  10. 2019Machine Unlearningcited by 1,061
  11. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  12. 2022Red Teaming Language Models with Language Modelscited by 1,026
16 more
  1. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  2. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  3. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  4. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  5. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
  6. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 163
  7. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 116
  8. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 114
  9. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 86
  10. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  11. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 65
  12. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  13. 2025Trading Inference-Time Compute for Adversarial Robustnesscited by 61
  14. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 45
  15. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 25
  16. 2024Revisiting the Robust Alignment of Circuit Breakerscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 1.

9 papers.

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 31
  3. 2025A Survey of Attacks on Large Language Modelscited by 15
  4. 2025Large Reasoning Models Are Autonomous Jailbreak Agentscited by 13
  5. 2025Toward a Dynamic Stackelberg Game-Theoretic Framework for Agentic AI Defense Against LLM Jailbreakingcited by 5
  6. 2025Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Modelscited by 1
  7. 2026Muse Spark Safety & Preparedness Reportcited by 1
  8. 2025Dark LLMs: The Growing Threat of Unaligned AI Modelscited by 0
  9. 2025Evaluating Multi-Agent Defences Against Jailbreaking Attacks on Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Hide Your Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Carrier Articles
  2. 2024Distract Large Language Models for Automatic Jailbreak Attack
  3. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks
  4. 2024Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring
  5. 2025Effective and Efficient Jailbreaks of Black-Box LLMs with Cross-Behavior Attacks
  6. 2024Rapid Response: Mitigating LLM Jailbreaks with a Few Examples
  7. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models
  8. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey