mlsanityData to 14 Oct 2026

Safety Alignment of LMs via Non-cooperative Games

arXiv 2512.20806 ↗First posted 23 Dec 2025cs.AIcited by 8

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Ensuring the safety of language models (LMs) while maintaining their usefulness remains a critical challenge in AI alignment. Current approaches rely on sequential adversarial training: generating adversarial prompts and fine-tuning LMs to defend against them. We introduce a different paradigm: framing safety alignment as a non-zero-sum game between an Attacker LM and a Defender LM trained jointly via online reinforcement learning. Each LM continuously adapts to the other's evolving strategies, driving iterative improvement. Our method uses a preference-based reward signal derived from pairwise comparisons instead of point-wise scores, providing more robust supervision and potentially reducing reward hacking. Our RL recipe, AdvGame, shifts the Pareto frontier of safety and utility, yielding a Defender LM that is simultaneously more helpful and more resilient to adversarial attacks. In addition, the resulting Attacker LM converges into a strong, general-purpose red-teaming agent that can be directly deployed to probe arbitrary target models. Code at github.com/facebookresearch/advgame.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 59 references that are in the map. First posted on arXiv 23 Dec 2025.

Builds on

40 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025The Alignment Waltz: Jointly Training Agents to Collaborate for Safety9 Oct
  2. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models9 Jun
  3. 2025Lifelong Safety Alignment for Language Models26 May
  4. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs20 Dec
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 35

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  12. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
23 more
  1. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  2. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  3. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  4. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  5. 2023Detecting Language Model Attacks with Perplexitycited by 431
  6. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  7. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  8. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  9. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  10. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  11. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  12. 2025Generalizing Verifiable Instruction Followingcited by 194
  13. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  14. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 157
  15. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 156
  16. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  17. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 45
  18. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 36
  19. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
  20. 2024Self-Improving Robust Preference Optimizationcited by 19
  21. 2023Evolving Diverse Red-team Language Models in Multi-round Multi-agent Gamescited by 17
  22. 2025Value-Free Policy Optimization via Reward Partitioningcited by 1
  23. 2025Strategic Deflection: Defending LLMs from Logit Manipulationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 21
  2. 2026Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPOcited by 5
  3. 2026Min-Max Optimization Requires Exponentially Many Queriescited by 1
  4. 2026Addressing Over-Refusal in LLMs with Competing Rewardscited by 0
  5. 2026Suan: Rectifying Direct Preference Safety Alignment in Large Language Modelscited by 0
  6. 2026Reflections and Fragments: Securing LLMs Against Sequential Mosaic Attackscited by 0
  7. 2026ReSI: Recursive Safety Improvement toward Resistant and Resilient AIcited by 0
  8. 2026AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safetycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Safety Game: Inference-Time Alignment of Black-Box LLMs via Constrained Optimization
  2. 2026TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignment
  3. 2025Agent Safety Alignment via Reinforcement Learning
  4. 2026Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replay
  5. 2025Adversarial Reinforcement Learning for Large Language Model Agent Safety
  6. 2024Toward Optimal LLM Alignments Using Two-Player Games
  7. 2026Alignment Risks from Capability-Seeking RL Training
  8. 2026Yuvion LLM: An Adversarially-Aware Large Language Model for Content And AI Safety