mlsanityData to 14 Oct 2026

Stable Opponent Shaping in Differentiable Games

arXiv 1811.08469 ↗First posted 20 Nov 2018cs.MA cs.AI cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

A growing number of learning methods are actually differentiable games whose players optimise multiple, interdependent objectives in parallel -- from GANs and intrinsic curiosity to multi-agent RL. Opponent shaping is a powerful approach to improve learning dynamics in these games, accounting for player influence on others' updates. Learning with Opponent-Learning Awareness (LOLA) is a recent algorithm that exploits this response and leads to cooperation in settings like the Iterated Prisoner's Dilemma. Although experimentally successful, we show that LOLA agents can exhibit 'arrogant' behaviour directly at odds with convergence. In fact, remarkably few algorithms have theoretical guarantees applying across all (n-player, non-convex) games. In this paper we present Stable Opponent Shaping (SOS), a new method that interpolates between LOLA and a stable variant named LookAhead. We prove that LookAhead converges locally to equilibria and avoids strict saddles in all differentiable games. SOS inherits these essential guarantees, while also shaping the learning of opponents and consistently either matching or outperforming LOLA experimentally.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 8; 2023: 10; 2024: 9; 2025: 3; 2026: 8.

49 papers.

Sort:

By area:

  1. 2022Model-Free Opponent Shapingcited by 44
  2. 2022COLA: Consistent Learning with Opponent-Learning Awarenesscited by 38
  3. 2023A Review of Cooperation in Multi-agent Learningcited by 29
  4. 2022Proximal Learning With Opponent-Learning Awarenesscited by 20
  5. 2021Stackelberg Actor-Critic: Game-Theoretic Reinforcement Learning Algorithmscited by 19
  6. 2022Influencing Long-Term Behavior in Multiagent Reinforcement Learningcited by 18
  7. 2021ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxationcited by 17
  8. 2022Adversarial Cheap Talkcited by 16
  9. 2021Cooperation and Reputation Dynamics with Reinforcement Learningcited by 13
  10. 2021Emergent Prosociality in Multi-Agent Games Through Giftingcited by 12
  11. 2021A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learningcited by 12
  12. 2023Scaling Opponent Shaping to High Dimensional Gamescited by 12
37 more
  1. 2024Multi-Agent, Human-Agent and Beyond: A Survey on Cooperation in Social Dilemmascited by 12
  2. 2023Incentivizing honest performative predictions with proper scoring rulescited by 9
  3. 2021Minimax Optimization with Smooth Algorithmic Adversariescited by 8
  4. 2021Lyapunov Exponents for Diversity in Differentiable Gamescited by 8
  5. 2024LOQA: Learning with Opponent Q-Learning Awarenesscited by 8
  6. 2021A Game-Theoretic Approach to Multi-Agent Trust Region Optimizationcited by 7
  7. 2024Aligning Individual and Collective Objectives in Multi-Agent Cooperationcited by 7
  8. 2024Advantage Alignment Algorithmscited by 7
  9. 2023Meta-Value Learning: a General Framework for Learning with Learning Awarenesscited by 6
  10. 2024Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Gamescited by 6
  11. 2021Dealing with Non-Stationarity in MARL via Trust-Region Decompositioncited by 5
  12. 2022Game-Theoretical Perspectives on Active Equilibria: A Preferred Solution Concept over Nash Equilibriacited by 5
  13. 2023(Ir)rationality in AI: State of the Art, Research Challenges and Open Questionscited by 5
  14. 2022Similarity-based cooperative equilibriumcited by 4
  15. 2024Multi-agent cooperation through learning-aware policy gradientscited by 4
  16. 2021Improving Social Welfare While Preserving Autonomy via a Pareto Mediatorcited by 3
  17. 2023IQ-Flow: Mechanism Design for Inducing Cooperative Behavior to Self-Interested Agents in Sequential Social Dilemmascited by 3
  18. 2023Learning to Influence Human Behavior with Offline Reinforcement Learningcited by 2
  19. 2024Analysing the Sample Complexity of Opponent Shapingcited by 2
  20. 2025Focusing Influence Mechanism for Multi-Agent Reinforcement Learningcited by 2
  21. 2025Opponent Shaping in LLM Agentscited by 2
  22. 2025Robust and Diverse Multi-Agent Learning via Rational Policy Gradientcited by 2
  23. 2023ApproxED: Approximate exploitability descent via learned best responsescited by 1
  24. 2026Min-Max Optimization Requires Exponentially Many Queriescited by 1
  25. 2021Behaviour-conditioned policies for cooperative reinforcement learning taskscited by 0
  26. 2022Learning Collective Action under Risk Diversitycited by 0
  27. 2023Off-Policy Action Anticipation in Multi-Agent Reinforcement Learningcited by 0
  28. 2023Leading the Pack: N-player Opponent Shapingcited by 0
  29. 2024Evolution of Societies via Reinforcement Learningcited by 0
  30. 2024Preference-based opponent shaping in differentiable gamescited by 0
  31. 2026Towards Sustainable Investment Policies Informed by Opponent Shapingcited by 0
  32. 2026HALO: Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimizationcited by 0
  33. 2026A Structural Threshold in Decision Capacity Governs Collapse in Self-Play Reinforcement Learningcited by 0
  34. 2026Equilibrium Selection in Multi-Agent Policy Gradients via Opponent-Aware Basin Entrycited by 0
  35. 2026Differentiable Belief-based Opponent Shapingcited by 0
  36. 2026Evolving in the Agent Jungle via History-Informed Opponent Awarenesscited by 0
  37. 2026Learning to Harvest Without Collapse in a Regenerative Commons: A Lagrangian Frameworkcited by 0