mlsanityData to 14 Oct 2026

Safe Equilibrium Policy Optimization for Strategic Agent Policies

arXiv 2605.30854 ↗First posted 29 May 2026cs.MA cs.AIcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Language models fine-tuned with reinforcement learning typically optimize for task reward, ignoring multi-agent strategic structure. Because these agents condition on natural language game-state descriptions and emit actions through free-form generation, strategic failure modes -- exploiting weaker opponents, coordinating on harmful equilibria, and externalizing costs are inseparable from the language interface itself. We propose Safe Equilibrium Policy Optimization (sepo), a training objective that augments expected payoff with explicit penalties for exploitability, collusion risk, and externality cost. We implement sepo as a reward signal for Group Relative Policy Optimization (GRPO), applied to Gemma~4 E4B-it and Qwen~3.5-4B after supervised fine-tuning (SFT). Evaluated across five strategic domains: Iterated Prisoner's Dilemma, repeated auctions, two negotiation variants, and Kuhn Poker. sepo achieves zero exploit-pool advantage in Kuhn Poker for both models, outperforms the base model on safety in four domains, and corrects the over-cooperative behavior introduced by SFT. In negotiation, sepo achieves a positive-safety outcome and only the positive normalized relative advantage of any negotiation configuration. Ablation experiments confirm that per-rollout exploit computation is necessary: a shared constant penalty cancels in GRPO advantage normalization (constant control-variate property), producing zero gradient. To support further research in strategic safety for agents, we release our hrefhttps://anonymous.4open.science/r/sepo-2668/README.mdcode and SFT datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 37 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

23 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-429 Sep

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  8. 2016Concrete Problems in AI Safetycited by 1,642
  9. 2017Constrained Policy Optimizationcited by 983
  10. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 268
  11. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 241
  12. 2017Learning with Opponent-Learning Awarenesscited by 222
10 more
  1. 2017Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 209
  2. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 209
  3. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 207
  4. 2018Decoupling Strategy and Generation in Negotiation Dialoguescited by 162
  5. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
  6. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 120
  7. 2018Inequity aversion improves cooperation in intertemporal social dilemmascited by 106
  8. 2018Deep Counterfactual Regret Minimizationcited by 103
  9. 2019Computing Approximate Equilibria in Sequential Adversarial Games by Exploitability Descentcited by 31
  10. 2024K-Level Reasoning: Establishing Higher Order Beliefs in Large Language Models for Strategic Reasoningcited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Implicit Strategic Optimization: Rethinking Long-Horizon Decision-Making in Adversarial Poker Environments
  2. 2026Same Game, Different Story: A Minimal Conservative Strategic Robustness Benchmark for Large Language Model Agents
  3. 2025Aligning Large Language Model Agents with Rational and Moral Preferences: A Supervised Fine-Tuning Approach
  4. 2026Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds
  5. 2025Policy-Conditioned Policies for Multi-Agent Task Solving
  6. 2025EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learning
  7. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models
  8. 2026Breaking the Impasse: Dual-Scale Evolutionary Policy Training for Social Language Agents