mlsanityData to 7 Oct 2026

Model-Free Opponent Shaping

arXiv 2205.01447 ↗First posted 3 May 2022cs.AI cs.MAcited by 43

Not on the map. Why?

Abstract (arXiv)

In general-sum games, the interaction of self-interested learning agents commonly leads to collectively worst-case outcomes, such as defect-defect in the iterated prisoner's dilemma (IPD). To overcome this, some methods, such as Learning with Opponent-Learning Awareness (LOLA), shape their opponents' learning process. However, these methods are myopic since only a small number of steps can be anticipated, are asymmetric since they treat other agents as naive learners, and require the use of higher-order derivatives, which are calculated through white-box access to an opponent's differentiable learning algorithm. To address these issues, we propose Model-Free Opponent Shaping (M-FOS). M-FOS learns in a meta-game in which each meta-step is an episode of the underlying inner game. The meta-state consists of the inner policies, and the meta-policy produces a new inner policy to be used in the next episode. M-FOS then uses generic model-free optimisation methods to learn meta-policies that accomplish long-horizon opponent shaping. Empirically, M-FOS near-optimally exploits naive learners and other, more sophisticated algorithms from the literature. For example, to the best of our knowledge, it is the first method to learn the well-known Zero-Determinant (ZD) extortion strategy in the IPD. In the same settings, M-FOS leads to socially optimal outcomes under meta-self-play. Finally, we show that M-FOS can be scaled to high-dimensional settings.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 18 of its 37 references that are in the map. First posted on arXiv 3 May 2022.

Builds on

18 of 37 references.

References · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2017Learning with Opponent-Learning Awarenesscited by 218
  4. 2017Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learningcited by 148
  5. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 106
  6. 2020Learning Latent Representations to Influence Multi-Agent Interactioncited by 92
  7. 2019Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learningcited by 68
  8. 2018Modeling Others using Oneself in Multi-Agent Reinforcement Learningcited by 64
  9. 2022HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learningcited by 55
  10. 2018Stable Opponent Shaping in Differentiable Gamescited by 49
  11. 2017Maintaining cooperation in complex social dilemmas using deep reinforcement learningcited by 48
  12. 2018DiCE: The Infinitely Differentiable Monte-Carlo Estimatorcited by 46
6 more
  1. 2019Differentiable Game Mechanicscited by 43
  2. 2020A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learningcited by 40
  3. 2022COLA: Consistent Learning with Opponent-Learning Awarenesscited by 38
  4. 2021Model-Based Opponent Modelingcited by 21
  5. 2017Reinforcement Learning Produces Dominant Strategies for the Iterated Prisoner's Dilemmacited by 7
  6. 2021On the Practical Consistency of Meta-Reinforcement Learning Algorithmscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 9; 2024: 18; 2025: 6; 2026: 5.

43 papers.

Sort:

By area:

  1. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  2. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 59
  3. 2023A Review of Cooperation in Multi-agent Learningcited by 27
  4. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  5. 2022Proximal Learning With Opponent-Learning Awarenesscited by 20
  6. 2024Discovering Temporally-Aware Reinforcement Learning Algorithmscited by 18
  7. 2022Adversarial Cheap Talkcited by 16
  8. 2022Oracles & Followers: Stackelberg Equilibria in Deep Multi-Agent Reinforcement Learningcited by 12
  9. 2024Multi-Agent, Human-Agent and Beyond: A Survey on Cooperation in Social Dilemmascited by 12
  10. 2024Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learningcited by 8
  11. 2023Cooperative AI via Decentralized Commitment Devicescited by 7
  12. 2024Aligning Individual and Collective Objectives in Multi-Agent Cooperationcited by 7
31 more
  1. 2024Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planningcited by 7
  2. 2024EvIL: Evolution Strategies for Generalisable Imitation Learningcited by 7
  3. 2024Advantage Alignment Algorithmscited by 7
  4. 2023Meta-Value Learning: a General Framework for Learning with Learning Awarenesscited by 6
  5. 2024JaxLife: An Open-Ended Agentic Simulatorcited by 6
  6. 2022Game-Theoretical Perspectives on Active Equilibria: A Preferred Solution Concept over Nash Equilibriacited by 5
  7. 2022Learning Latent Representations to Co-Adapt to Humanscited by 5
  8. 2023(Ir)rationality in AI: State of the Art, Research Challenges and Open Questionscited by 5
  9. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environmentscited by 5
  10. 2023Arbitrary Order Meta-Learning with Simple Population-Based Evolutioncited by 4
  11. 2024Normative Modules: A Generative Agent Architecture for Learning Norms that Supports Multi-Agent Cooperationcited by 4
  12. 2024Crafting Desirable Climate Trajectories with RL Explored Socio-Environmental Simulationscited by 4
  13. 2024Multi-agent cooperation through learning-aware policy gradientscited by 4
  14. 2024Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactionscited by 3
  15. 2024Reciprocal Reward Influence Encourages Cooperation From Self-Interested Agentscited by 2
  16. 2024Learning to Steer Markovian Agents under Model Uncertaintycited by 2
  17. 2025Opponent Shaping in LLM Agentscited by 2
  18. 2025Robust and Diverse Multi-Agent Learning via Rational Policy Gradientcited by 2
  19. 2024ADIOS: Antibody Development via Opponent Shapingcited by 1
  20. 2026Multi-agent cooperation through in-context co-player inferencecited by 1
  21. 2023Uncoupled Learning of Differential Stackelberg Equilibria with Commitmentscited by 0
  22. 2023Off-Policy Action Anticipation in Multi-Agent Reinforcement Learningcited by 0
  23. 2023Leading the Pack: N-player Opponent Shapingcited by 0
  24. 2024Evolution of Societies via Reinforcement Learningcited by 0
  25. 2024Preference-based opponent shaping in differentiable gamescited by 0
  26. 2025Fast Adaptive Anti-Jamming Channel Access via Deep Q Learning and Coarse-Grained Spectrum Predictioncited by 0
  27. 2025Single-Agent Scaling Fails Multi-Agent Intelligence: Towards Foundation Models with Native Multi-Agent Intelligencecited by 0
  28. 2026Differentiable Belief-based Opponent Shapingcited by 0
  29. 2026Generalized Intention Modeling in Multi-Agent Reinforcement Learningcited by 0
  30. 2026Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasonerscited by 0
  31. 2026RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experimentscited by 0