mlsanityData to 14 Oct 2026

Toward Optimal LLM Alignments Using Two-Player Games

arXiv 2406.10977 ↗First posted 16 Jun 2024cs.CL cs.AIcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The standard Reinforcement Learning from Human Feedback (RLHF) framework primarily focuses on optimizing the performance of large language models using pre-collected prompts. However, collecting prompts that provide comprehensive coverage is both tedious and challenging, and often fails to include scenarios that LLMs need to improve on the most. In this paper, we investigate alignment through the lens of two-agent games, involving iterative interactions between an adversarial and a defensive agent. The adversarial agent's task at each step is to generate prompts that expose the weakness of the defensive agent. In return, the defensive agent seeks to improve its responses to these newly identified prompts it struggled with, based on feedback from the reward model. We theoretically demonstrate that this iterative reinforcement learning optimization converges to a Nash Equilibrium for the game induced by the agents. Experimental results in safety scenarios demonstrate that learning in such a competitive environment not only fully trains agents but also leads to policies with enhanced generalization capabilities for both adversarial and defensive agents.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 54 of its 66 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

54 of 66 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  2. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  3. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation8 Mar
  4. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  5. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Data17 Dec
  6. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  7. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  11. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 42

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
30 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  2. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  3. 2022Red Teaming Language Models with Language Modelscited by 1,026
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  7. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  8. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  9. 2020Aligning AI With Shared Human Valuescited by 723
  10. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  11. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  12. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  13. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  14. 2023Detecting Language Model Attacks with Perplexitycited by 431
  15. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  16. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  17. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 417
  18. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  19. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 247
  20. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
  21. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  22. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  23. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  24. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  25. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 163
  26. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 129
  27. 2017Emergent Complexity via Multi-Agent Competitioncited by 120
  28. 2020Near-Optimal Reinforcement Learning with Self-Playcited by 109
  29. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 88
  30. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 83

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 5.

13 papers.

Sort:

By area:

  1. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  2. 2025MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teamingcited by 32
  3. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  4. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 21
  5. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 16
  6. 2025Frictional Agent Alignment Framework: Slow Down and Don't Break Thingscited by 10
  7. 2025The Alignment Waltz: Jointly Training Agents to Collaborate for Safetycited by 10
  8. 2025Improving Rationality in the Reasoning Process of Language Models through Self-playing Gamecited by 7
  9. 2024Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimizationcited by 1
  10. 2026Exploring Weaknesses in Function Call Models via Reinforcement Learning: An Adversarial Data Augmentation Approachcited by 0
  11. 2026Model-Agnostic Lifelong LLM Safety via Externalized Attack-Defense Co-Evolutioncited by 0
  12. 2026All you need to break LLMs are Black-Box, Adapting, Efficient, Transferable, Harmful, Applicable ... Attackscited by 0
1 more
  1. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
  2. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
  3. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
  4. 2024Large Language Models as Agents in Two-Player Games
  5. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  6. 2025Safety Alignment of LMs via Non-cooperative Games
  7. 2024True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learning
  8. 2024Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting