mlsanityData to 14 Oct 2026

TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewards

arXiv 2512.07761 ↗First posted 8 Dec 2025cs.AI cs.LGcited by 14

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language models have seen widespread adoption, yet they remain vulnerable to multi-turn jailbreak attacks, threatening their safe deployment. This has led to the task of training automated multi-turn attackers to probe model safety vulnerabilities. However, existing approaches typically rely on turn-level optimization, which is insufficient for learning long-term attack strategies. To bridge this gap, we formulate this task as a multi-turn reinforcement learning problem, directly optimizing the harmfulness of the final-turn response as the outcome reward. To address the sparse supervision of the outcome reward, we introduce TROJail, which employs two process rewards to evaluate the utility of intermediate prompts and integrate them into advantage estimation. These rewards (1) penalize overly harmful prompts that trigger the model's refusal mechanism, and (2) encourage steering the semantic relevance of responses toward the targeted harmful content. Experimental results show improved attack success rates across multiple models and benchmarks, highlighting the effectiveness of our approach. The code is available at https://github.com/xxiqiao/TROJail. Warning: This paper contains examples of harmful content.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 36 of its 55 references that are in the map. First posted on arXiv 8 Dec 2025.

Builds on

36 of 55 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Stand on The Shoulders of Giants: Building JailExpert from Previous Attack Experience25 Aug
  2. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs23 May
  3. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents15 Apr
  4. 2024MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue6 Nov
  5. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs3 Oct
  6. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet27 Aug
  7. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack2 Apr
  8. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  9. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  11. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 24

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  6. 2022Red Teaming Language Models with Language Modelscited by 1,026
  7. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  8. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  9. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  10. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  11. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
  12. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
12 more
  1. 2024Free Process Rewards without Process Labelscited by 165
  2. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 78
  3. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 77
  4. 2024Direct Multi-Turn Preference Optimization for Language Agentscited by 64
  5. 2025Safety in Large Reasoning Models: A Surveycited by 64
  6. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  7. 2025SPEED: Scalable, Precise, and Efficient Concept Erasure for Diffusion Modelscited by 36
  8. 2025MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teamingcited by 32
  9. 2023nach0: Multimodal Natural and Chemical Languages Foundation Modelcited by 26
  10. 2024Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interactioncited by 25
  11. 2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learningcited by 15
  12. 2025IAG: Input-aware Backdoor Attack on VLM-based Visual Groundingcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 11
  2. 2026TRACE: Task-Aware Adaptive Self-Evolving Agentic Jailbreakingcited by 7
  3. 2026Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPOcited by 5
  4. 2026Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreakingcited by 3
  5. 2026Internalizing Safety Understanding in Large Reasoning Models via Verificationcited by 3
  6. 2026Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agentscited by 3
  7. 2026SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreakscited by 1
  8. 2026PsychJail: Exploring Psychological Jailbreaks via Multi-Turn Persuasion of LLM Policiescited by 1
  9. 2026Model-Agnostic Lifelong LLM Safety via Externalized Attack-Defense Co-Evolutioncited by 0
  10. 2026MJ: Multi-turn LLM Jailbreaking via Decomposed Credit Assignmentcited by 0
  11. 2026In-Context Learning as Implicit Policy Gradientcited by 0
  12. 2026TRACE: Trajectory Aware Reasoning for Multi-Turn Adversarial Conversation Evaluationcited by 0
2 more
  1. 2026HASTE: Evolving Agent Harnesses Against Emerging Attacks Using Sparse Evidencecited by 0
  2. 2026ATTCK-R: Knowledge-Aware Long-Horizon Jailbreaking for Protected LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026THRD: A Training-Free Multi-Turn Defense Framework for Jailbreak Attacks on Large Language Models
  2. 2026TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Models
  3. 2025GRAF: Multi-turn Jailbreaking via Global Refinement and Active Fabrication
  4. 2024PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak Approach
  5. 2025AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models
  6. 2025Breaking the Ceiling: Exploring the Potential of Jailbreak Attacks through Expanding Strategy Space
  7. 2025AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models
  8. 2025Multi-Turn Jailbreaks Are Simpler Than They Seem