mlsanityData to 7 Oct 2026

Model-Based Opponent Modeling

arXiv 2108.01843 ↗First posted 4 Aug 2021cs.LG cs.AI cs.MAcited by 21

Not on the map. Why?

Abstract (arXiv)

When one agent interacts with a multi-agent environment, it is challenging to deal with various opponents unseen before. Modeling the behaviors, goals, or beliefs of opponents could help the agent adjust its policy to adapt to different opponents. In addition, it is also important to consider opponents who are learning simultaneously or capable of reasoning. However, existing work usually tackles only one of the aforementioned types of opponents. In this paper, we propose model-based opponent modeling (MBOM), which employs the environment model to adapt to all kinds of opponents. MBOM simulates the recursive reasoning process in the environment model and imagines a set of improving opponent policies. To effectively and accurately represent the opponent policy, MBOM further mixes the imagined opponent policies according to the similarity with the real behaviors of opponents. Empirically, we show that MBOM achieves more effective adaptation than existing methods in a variety of tasks, respectively with different types of opponents, i.e., fixed policy, na"ive learner, and reasoning learner.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 4 Aug 2021.

Builds on

38 of 51 references.

References · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  5. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  8. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  9. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  10. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  11. 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 257
  12. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
26 more
  1. 2018Machine Theory of Mindcited by 222
  2. 2017Learning with Opponent-Learning Awarenesscited by 218
  3. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  4. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  5. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  6. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  7. 2017Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problemscited by 126
  8. 2017Value Prediction Networkcited by 117
  9. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 106
  10. 2016Opponent Modeling in Deep Reinforcement Learningcited by 94
  11. 2020Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexitycited by 91
  12. 2019Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learningcited by 68
  13. 2018Modeling Others using Oneself in Multi-Agent Reinforcement Learningcited by 64
  14. 2017Maintaining cooperation in complex social dilemmas using deep reinforcement learningcited by 48
  15. 2018Learning Policy Representations in Multiagent Systemscited by 48
  16. 2018DiCE: The Infinitely Differentiable Monte-Carlo Estimatorcited by 46
  17. 2020A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learningcited by 40
  18. 2021Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomizationcited by 35
  19. 2017A Deep Policy Inference Q-Network for Multi-Agent Systemscited by 24
  20. 2019Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoningcited by 24
  21. 2021Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rolloutscited by 24
  22. 2019Multi-Agent Reinforcement Learning with Multi-Step Generative Modelscited by 18
  23. 2020Variational Autoencoders for Opponent Modeling in Multi-Agent Systemscited by 18
  24. 2020Emergence of Pragmatics from Referential Game between Theory of Mind Agentscited by 17
  25. 2021MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world modelscited by 16
  26. 2020R2-B2: Recursive Reasoning-Based Bayesian Optimization for No-Regret Learning in Gamescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 3; 2024: 3; 2025: 7; 2026: 4.

21 papers.

Sort:

By area:

  1. 2022Model-Free Opponent Shapingcited by 43
  2. 2022Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospectscited by 17
  3. 2024Fast Peer Adaptation with Context-aware Explorationcited by 7
  4. 2022CLAS: Coordinating Multi-Robot Manipulation with Central Latent Action Spacescited by 6
  5. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decompositioncited by 4
  6. 2022Offline Equilibrium Findingcited by 2
  7. 2023Adversarial Search and Tracking with Multiagent Reinforcement Learning in Sparsely Observable Environmentcited by 2
  8. 2024Reciprocal Reward Influence Encourages Cooperation From Self-Interested Agentscited by 2
  9. 2025Trajectory-Class-Aware Multi-Agent Reinforcement Learningcited by 2
  10. 2023Model-Based Decentralized Policy Optimizationcited by 1
  11. 2023Fact-based Agent modeling for Multi-Agent Reinforcement Learningcited by 1
  12. 2024Mastering Zero-Shot Interactions in Cooperative and Competitive Simultaneous Gamescited by 1
9 more
  1. 2025Unifying Agent Interaction and World Information for Multi-agent Coordinationcited by 1
  2. 2026Robots Influencing Humans to Reveal their Goals during Collaboration and Competitioncited by 1
  3. 2025Fast Adaptive Anti-Jamming Channel Access via Deep Q Learning and Coarse-Grained Spectrum Predictioncited by 0
  4. 2025Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Reviewcited by 0
  5. 2025Bayesian Decision Making around Expertscited by 0
  6. 2025Graph Neural Networks, Deep Reinforcement Learning and Probabilistic Topic Modeling for Strategic Multiagent Settingscited by 0
  7. 2026OpenGuanDan: A Large-Scale Imperfect Information Game Benchmarkcited by 0
  8. 2026Automated Approach for Solving Infinite-state Polynomial Reachability Gamescited by 0
  9. 2026Differentiable Belief-based Opponent Shapingcited by 0