# A Deep Policy Inference Q-Network for Multi-Agent Systems

arXiv: https://arxiv.org/abs/1712.07893 · HTML: https://mlsanity.com/paper/1712.07893/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2017.

## Built on by (most cited first, 25 of 25)

- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [Continuous Coordination As a Realistic Scenario for Lifelong Learning](https://mlsanity.com/paper/2103.03216/) (2021)
- [Model-Based Opponent Modeling](https://mlsanity.com/paper/2108.01843/) (2021)
- [Collaborative Visual Navigation](https://mlsanity.com/paper/2107.01151/) (2021)
- [LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates](https://mlsanity.com/paper/2109.12508/) (2021)
- [Learning Altruistic Behaviours in Reinforcement Learning without External Rewards](https://mlsanity.com/paper/2107.09598/) (2021)
- [Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutions](https://mlsanity.com/paper/2409.13144/) (2024)
- [The Curse of Diversity in Ensemble-Based Exploration](https://mlsanity.com/paper/2405.04342/) (2024)
- [Multi-Agent Advisor Q-Learning](https://mlsanity.com/paper/2111.00345/) (2021)
- [Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learning](https://mlsanity.com/paper/2203.12048/) (2022)
- [BRExIt: On Opponent Modelling in Expert Iteration](https://mlsanity.com/paper/2206.00113/) (2022)
- [Offline Equilibrium Finding](https://mlsanity.com/paper/2207.05285/) (2022)
- [Decision-making with Speculative Opponent Models](https://mlsanity.com/paper/2211.11940/) (2022)
- [Metric Policy Representations for Opponent Modeling](https://mlsanity.com/paper/2106.05802/) (2021)
- [Fact-based Agent modeling for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2310.12290/) (2023)
- [Belief States for Cooperative Multi-Agent Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2504.08417/) (2025)
- [Learning Policy Representations for Steerable Behavior Synthesis](https://mlsanity.com/paper/2601.22350/) (2026)
- [Off-Policy Action Anticipation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.01447/) (2023)
- [Adaptive Opponent Policy Detection in Multi-Agent MDPs: Real-Time Strategy Switch Identification Using Running Error Estimation](https://mlsanity.com/paper/2406.06500/) (2024)
- [Effective Reward Specification in Deep Reinforcement Learning](https://mlsanity.com/paper/2412.07177/) (2024)
- [Fast Adaptive Anti-Jamming Channel Access via Deep Q Learning and Coarse-Grained Spectrum Prediction](https://mlsanity.com/paper/2502.04963/) (2025)
- [Scalable Multiagent Reinforcement Learning with Collective Influence Estimation](https://mlsanity.com/paper/2601.08210/) (2026)
- [Scaling up Energy-Aware Multi-Agent Reinforcement Learning for Mission-Oriented Drone Networks with Individual Reward](https://mlsanity.com/paper/2605.24992/) (2026)
- [When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2610.18229/) (2026)

## Abstract (arXiv)

We present DPIQN, a deep policy inference Q-network that targets multi-agent systems composed of controllable agents, collaborators, and opponents that interact with each other. We focus on one challenging issue in such systems---modeling agents with varying strategies---and propose to employ "policy features" learned from raw observations (e.g., raw images) of collaborators and opponents by inferring their policies. DPIQN incorporates the learned policy features as a hidden vector into its own deep Q-network (DQN), such that it is able to predict better Q values for the controllable agents than the state-of-the-art deep reinforcement learning models. We further propose an enhanced version of DPIQN, called deep recurrent policy inference Q-network (DRPIQN), for handling partial observability. Both DPIQN and DRPIQN are trained by an adaptive training procedure, which adjusts the network's attention to learn the policy features and its own Q-values at different phases of the training process. We present a comprehensive analysis of DPIQN and DRPIQN, and highlight their effectiveness and generalizability in various multi-agent settings. Our models are evaluated in a classic soccer game involving both competitive and collaborative scenarios. Experimental results performed on 1 vs. 1 and 2 vs. 2 games show that DPIQN and DRPIQN demonstrate superior performance to the baseline DQN and deep recurrent Q-network (DRQN) models. We also explore scenarios in which collaborators or opponents dynamically change their policies, and show that DPIQN and DRPIQN do lead to better overall performance in terms of stability and mean scores.