# Opponent Modeling in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1609.05559 · HTML: https://mlsanity.com/paper/1609.05559/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2016.

## Built on by (most cited first, 30 of 97)

- [A Survey on Model-based Reinforcement Learning](https://mlsanity.com/paper/2206.09328/) (2022)
- [When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learning](https://mlsanity.com/paper/2203.16797/) (2022)
- [ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind](https://mlsanity.com/paper/2111.09189/) (2021)
- [NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned](https://mlsanity.com/paper/2101.00133/) (2021)
- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [COLA: Consistent Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/2203.04098/) (2022)
- [The Neural Race Reduction: Dynamics of Abstraction in Gated Networks](https://mlsanity.com/paper/2207.10430/) (2022)
- [Dynamic Handover: Throw and Catch with Bimanual Hands](https://mlsanity.com/paper/2309.05655/) (2023)
- [Deep Interactive Bayesian Reinforcement Learning via Meta-Learning](https://mlsanity.com/paper/2101.03864/) (2021)
- [PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2301.06387/) (2023)
- [Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts](https://mlsanity.com/paper/2105.03363/) (2021)
- [Influencing Towards Stable Multi-Agent Interactions](https://mlsanity.com/paper/2110.08229/) (2021)
- [PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning](https://mlsanity.com/paper/2102.12560/) (2021)
- [A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios](https://mlsanity.com/paper/2412.03920/) (2024)
- [Model-Based Opponent Modeling](https://mlsanity.com/paper/2108.01843/) (2021)
- [Theory of Mind as Intrinsic Motivation for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2307.01158/) (2023)
- [Influencing Long-Term Behavior in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2203.03535/) (2022)
- [Coordination with Humans via Strategy Matching](https://mlsanity.com/paper/2210.15099/) (2022)
- [LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates](https://mlsanity.com/paper/2109.12508/) (2021)
- [TiAda: A Time-scale Adaptive Algorithm for Nonconvex Minimax Optimization](https://mlsanity.com/paper/2210.17478/) (2022)
- [DouZero+: Improving DouDizhu AI by Opponent Modeling and Coach-guided Learning](https://mlsanity.com/paper/2204.02558/) (2022)
- [DanZero+: Dominating the GuanDan Game through Reinforcement Learning](https://mlsanity.com/paper/2312.02561/) (2023)
- [The Curse of Diversity in Ensemble-Based Exploration](https://mlsanity.com/paper/2405.04342/) (2024)
- [Multi-Agent Inverse Reinforcement Learning in Real World Unstructured Pedestrian Crowds](https://mlsanity.com/paper/2405.16439/) (2024)
- [Investigation of Independent Reinforcement Learning Algorithms in Multi-Agent Environments](https://mlsanity.com/paper/2111.01100/) (2021)
- [DanZero: Mastering GuanDan Game with Reinforcement Learning](https://mlsanity.com/paper/2210.17087/) (2022)
- [iPLAN: Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.06236/) (2023)
- [Learning Models of Adversarial Agent Behavior under Partial Observability](https://mlsanity.com/paper/2306.11168/) (2023)
- [(Ir)rationality in AI: State of the Art, Research Challenges and Open Questions](https://mlsanity.com/paper/2311.17165/) (2023)
- [GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration](https://mlsanity.com/paper/2502.19684/) (2025)

## Abstract (arXiv)

Opponent modeling is necessary in multi-agent settings where secondary agents with competing goals also adapt their strategies, yet it remains challenging because strategies interact with each other and change. Most previous work focuses on developing probabilistic models or parameterized strategies for specific applications. Inspired by the recent success of deep reinforcement learning, we present neural-based models that jointly learn a policy and the behavior of opponents. Instead of explicitly predicting the opponent's action, we encode observation of the opponents into a deep Q-Network (DQN); however, we retain explicit modeling (if desired) using multitasking. By using a Mixture-of-Experts architecture, our model automatically discovers different strategy patterns of opponents without extra supervision. We evaluate our models on a simulated soccer game and a popular trivia game, showing superior performance over DQN and its variants.