mlsanityData to 14 Oct 2026

Scaling Opponent Shaping to High Dimensional Games

arXiv 2312.12568 ↗First posted 19 Dec 2023cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

In multi-agent settings with mixed incentives, methods developed for zero-sum games have been shown to lead to detrimental outcomes. To address this issue, opponent shaping (OS) methods explicitly learn to influence the learning dynamics of co-players and empirically lead to improved individual and collective outcomes. However, OS methods have only been evaluated in low-dimensional environments due to the challenges associated with estimating higher-order derivatives or scaling model-free meta-learning. Alternative methods that scale to more complex settings either converge to undesirable solutions or rely on unrealistic assumptions about the environment or co-players. In this paper, we successfully scale an OS-based approach to general-sum games with temporally-extended actions and long-time horizons for the first time. After analysing the representations of the meta-state and history used by previous algorithms, we propose a simplified version called Shaper. We show empirically that Shaper leads to improved individual and collective outcomes in a range of challenging settings from literature. We further formalize a technique previously implicit in the literature, and analyse its contribution to opponent shaping. We show empirically that this technique is helpful for the functioning of prior methods in certain environments. Lastly, we show that previous environments, such as the CoinGame, are inadequate for analysing temporally-extended general-sum interactions.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 20 of its 47 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

20 of 47 references.

References · 20

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2019The StarCraft Multi-Agent Challengecited by 597
  3. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  4. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  5. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 264
  6. 2017Learning with Opponent-Learning Awarenesscited by 222
  7. 2020Open Problems in Cooperative AIcited by 168
  8. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 108
  9. 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 99
  10. 2018Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learningcited by 68
  11. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 63
  12. 2018Stable Opponent Shaping in Differentiable Gamescited by 49
8 more
  1. 2019Differentiable Game Mechanicscited by 43
  2. 2020A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learningcited by 41
  3. 2022evosax: JAX-based Evolution Strategiescited by 41
  4. 2022COLA: Consistent Learning with Opponent-Learning Awarenesscited by 38
  5. 2022Proximal Learning With Opponent-Learning Awarenesscited by 20
  6. 2020Model-free conventions in multi-agent reinforcement learning with heterogeneous preferencescited by 13
  7. 2022The Good Shepherd: An Oracle Agent for Mechanism Designcited by 10
  8. 2017Reinforcement Learning Produces Dominant Strategies for the Iterated Prisoner's Dilemmacited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 3; 2026: 2.

12 papers.

  1. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  2. 2024EvIL: Evolution Strategies for Generalisable Imitation Learningcited by 8
  3. 2024Advantage Alignment Algorithmscited by 7
  4. 2024Normative Modules: A Generative Agent Architecture for Learning Norms that Supports Multi-Agent Cooperationcited by 4
  5. 2024Multi-agent cooperation through learning-aware policy gradientscited by 4
  6. 2024Reciprocal Reward Influence Encourages Cooperation From Self-Interested Agentscited by 3
  7. 2025Opponent Shaping in LLM Agentscited by 2
  8. 2025Robust and Diverse Multi-Agent Learning via Rational Policy Gradientcited by 2
  9. 2025K-Level Policy Gradients for Multi-Agent Reinforcement Learningcited by 1
  10. 2026Multi-agent cooperation through in-context co-player inferencecited by 1
  11. 2024Environment Complexity and Nash Equilibria in a Sequential Social Dilemmacited by 0
  12. 2026RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experimentscited by 0