mlsanityData to 7 Oct 2026

Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

arXiv 2506.07468 ↗First posted 9 Jun 2025cs.LG cs.CL cs.MAcited by 33

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 2 more
Abstract (arXiv)

Conventional large language model (LLM) safety alignment relies on a reactive, disjoint loop: attackers exploit a static model, then defenders patch exposed vulnerabilities. This sequential setup leads to attackers overfitting obsolete exploits while defenders perpetually lag behind emerging threats. To address this, we introduce Self-RedTeam, the first fully online self-play multi-agent reinforcement learning (MARL) algorithm that continuously co-evolves attacker and defender for robust safety alignment. A single policy self-plays as both attacker and defender, generating adversarial prompts and defending against them, with a reward model adjudicating outcomes. Each role uses hidden chain-of-thought for strategic planning. Grounded in two-player zero-sum game theory, we establish a theoretical safety guarantee: if the game converges to Nash Equilibrium, the defender produces safe responses against any adversarial input. Empirically, Self-RedTeam generalizes across five models from the Llama and Qwen families, uncovering more diverse attacks (+17.80% SBERT) and improving safety of RLHF-trained models by up to 95% across 14 benchmarks. Our work motivates a shift from reactive patching to proactive co-evolution, enabling LLM safety self-improvement via online self-play MARL. Link to code: https://github.com/mickelliu/selfplay-redteaming

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 74 of its 114 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

74 of 114 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  5. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 68

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
38 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  8. 2022Red Teaming Language Models with Language Modelscited by 983
  9. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  11. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  12. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  13. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  14. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  15. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  16. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  17. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  18. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  19. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  20. 2024A StrongREJECT for Empty Jailbreakscited by 352
  21. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  22. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  23. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  24. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  25. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  26. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  27. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 233
  28. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  29. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
  30. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  31. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  32. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 172
  33. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  34. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 144
  35. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  36. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  37. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  38. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 22.

33 papers.

Sort:

By area:

  1. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  2. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 78
  3. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolutioncited by 58
  4. 2025Self-Questioning Language Modelscited by 47
  5. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 31
  6. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 23
  7. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  8. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 15
  9. 2025The Alignment Waltz: Jointly Training Agents to Collaborate for Safetycited by 10
  10. 2025Towards Understanding Self-play for LLM Reasoningcited by 8
  11. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Criticcited by 7
  12. 2026ProbeLLM: Automating Principled Diagnosis of LLM Failurescited by 7
21 more
  1. 2025Safety Alignment of LMs via Non-cooperative Gamescited by 6
  2. 2026Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replaycited by 6
  3. 2025Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learningcited by 5
  4. 2026GPT-Red: Automated Red Teaming via Self-Play at Scalecited by 5
  5. 2025Mitigating Over-Refusal in Aligned Large Language Models via Inference-Time Activation Energycited by 4
  6. 2026AgenticRed: Evolving Agentic Systems for Red-Teamingcited by 4
  7. 2026Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPOcited by 4
  8. 2026Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMscited by 3
  9. 2026Profit is the Red Team: Stress-Testing Agents in Strategic Economic Interactionscited by 1
  10. 2026Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Designcited by 1
  11. 2026One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RLcited by 1
  12. 2026Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routingcited by 0
  13. 2026Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of Mindcited by 0
  14. 2026Disentangling Intent from Role: Adversarial Self-Play for Persona-Invariant Safety Alignmentcited by 0
  15. 2026The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Playcited by 0
  16. 2026INFUSER: Influence-Guided Self-Evolution Improves Reasoningcited by 0
  17. 2026Addressing Over-Refusal in LLMs with Competing Rewardscited by 0
  18. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0
  19. 2026ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testingcited by 0
  20. 2026Divide, Consult, Conquer: Capability Laundering Through Aligned LLMscited by 0
  21. 2026ReSI: Recursive Safety Improvement toward Resistant and Resilient AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Adversarial Reinforcement Learning for Large Language Model Agent Safety
  2. 2026TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignment
  3. 2025Agent Safety Alignment via Reinforcement Learning
  4. 2025AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning
  5. 2025Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safety
  6. 2026ARES: Adaptive Red-Teaming and End-to-End Repair of Policy-Reward System
  7. 2026CHASE: Adversarial Red-Blue Teaming for Improving LLM Safety using Reinforcement Learning
  8. 2024Toward Optimal LLM Alignments Using Two-Player Games