# Model-Based Opponent Modeling

arXiv: https://arxiv.org/abs/2108.01843 · HTML: https://mlsanity.com/paper/2108.01843/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 51 references that are in the map.
- First posted on arXiv 4 Aug 2021.

## Builds on (most cited first, 30 of 38)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Model-Ensemble Trust-Region Policy Optimization](https://mlsanity.com/paper/1802.10592/) (2018)
- [Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion](https://mlsanity.com/paper/1807.01675/) (2018)
- [Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](https://mlsanity.com/paper/1803.00101/) (2018)
- [Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees](https://mlsanity.com/paper/1807.03858/) (2018)
- [Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems](https://mlsanity.com/paper/1709.08071/) (2017)
- [Value Prediction Network](https://mlsanity.com/paper/1707.03497/) (2017)
- [Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments](https://mlsanity.com/paper/1710.03641/) (2017)
- [Opponent Modeling in Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05559/) (2016)
- [Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexity](https://mlsanity.com/paper/2007.07461/) (2020)
- [Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1901.09207/) (2019)
- [Modeling Others using Oneself in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.09640/) (2018)
- [Maintaining cooperation in complex social dilemmas using deep reinforcement learning](https://mlsanity.com/paper/1707.01068/) (2017)
- [Learning Policy Representations in Multiagent Systems](https://mlsanity.com/paper/1806.06464/) (2018)
- [DiCE: The Infinitely Differentiable Monte-Carlo Estimator](https://mlsanity.com/paper/1802.05098/) (2018)
- [A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2011.00382/) (2020)
- [Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization](https://mlsanity.com/paper/2103.04564/) (2021)

## Built on by (most cited first, 22 of 22)

- [Model-Free Opponent Shaping](https://mlsanity.com/paper/2205.01447/) (2022)
- [Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospects](https://mlsanity.com/paper/2203.10603/) (2022)
- [Fast Peer Adaptation with Context-aware Exploration](https://mlsanity.com/paper/2402.02468/) (2024)
- [CLAS: Coordinating Multi-Robot Manipulation with Central Latent Action Spaces](https://mlsanity.com/paper/2211.15824/) (2022)
- [Offline Multi-agent Reinforcement Learning via Sequential Score Decomposition](https://mlsanity.com/paper/2505.05968/) (2025)
- [Reciprocal Reward Influence Encourages Cooperation From Self-Interested Agents](https://mlsanity.com/paper/2406.01641/) (2024)
- [Offline Equilibrium Finding](https://mlsanity.com/paper/2207.05285/) (2022)
- [Model-Based Decentralized Policy Optimization](https://mlsanity.com/paper/2302.08139/) (2023)
- [Adversarial Search and Tracking with Multiagent Reinforcement Learning in Sparsely Observable Environment](https://mlsanity.com/paper/2306.11301/) (2023)
- [Trajectory-Class-Aware Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.01440/) (2025)
- [Unifying Agent Interaction and World Information for Multi-agent Coordination](https://mlsanity.com/paper/2509.25550/) (2025)
- [Fact-based Agent modeling for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2310.12290/) (2023)
- [Mastering Zero-Shot Interactions in Cooperative and Competitive Simultaneous Games](https://mlsanity.com/paper/2402.03136/) (2024)
- [Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Review](https://mlsanity.com/paper/2507.10142/) (2025)
- [Robots Influencing Humans to Reveal their Goals during Collaboration and Competition](https://mlsanity.com/paper/2609.05519/) (2026)
- [Fast Adaptive Anti-Jamming Channel Access via Deep Q Learning and Coarse-Grained Spectrum Prediction](https://mlsanity.com/paper/2502.04963/) (2025)
- [Bayesian Decision Making around Experts](https://mlsanity.com/paper/2510.08113/) (2025)
- [Graph Neural Networks, Deep Reinforcement Learning and Probabilistic Topic Modeling for Strategic Multiagent Settings](https://mlsanity.com/paper/2511.10501/) (2025)
- [Omniscient Attacker in Stochastic Security Games with Interdependent Nodes](https://mlsanity.com/paper/2512.04561/) (2025)
- [OpenGuanDan: A Large-Scale Imperfect Information Game Benchmark](https://mlsanity.com/paper/2602.00676/) (2026)
- [Automated Approach for Solving Infinite-state Polynomial Reachability Games](https://mlsanity.com/paper/2605.10169/) (2026)
- [Differentiable Belief-based Opponent Shaping](https://mlsanity.com/paper/2605.29042/) (2026)

## Abstract (arXiv)

When one agent interacts with a multi-agent environment, it is challenging to deal with various opponents unseen before. Modeling the behaviors, goals, or beliefs of opponents could help the agent adjust its policy to adapt to different opponents. In addition, it is also important to consider opponents who are learning simultaneously or capable of reasoning. However, existing work usually tackles only one of the aforementioned types of opponents. In this paper, we propose model-based opponent modeling (MBOM), which employs the environment model to adapt to all kinds of opponents. MBOM simulates the recursive reasoning process in the environment model and imagines a set of improving opponent policies. To effectively and accurately represent the opponent policy, MBOM further mixes the imagined opponent policies according to the similarity with the real behaviors of opponents. Empirically, we show that MBOM achieves more effective adaptation than existing methods in a variety of tasks, respectively with different types of opponents, i.e., fixed policy, na"ive learner, and reasoning learner.