# Metric Policy Representations for Opponent Modeling

arXiv: https://arxiv.org/abs/2106.05802 · HTML: https://mlsanity.com/paper/2106.05802/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [QPLEX: Duplex Dueling Multi-Agent Q-Learning](https://mlsanity.com/paper/2008.01062/) (2020)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Learning Attentional Communication for Multi-Agent Cooperation](https://mlsanity.com/paper/1805.07733/) (2018)
- [Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems](https://mlsanity.com/paper/1709.08071/) (2017)
- [Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments](https://mlsanity.com/paper/1710.03641/) (2017)
- [Generative Modeling using the Sliced Wasserstein Distance](https://mlsanity.com/paper/1803.11188/) (2018)
- [Opponent Modeling in Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05559/) (2016)
- [Modeling Others using Oneself in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.09640/) (2018)
- [Learning Actionable Representations with Goal-Conditioned Policies](https://mlsanity.com/paper/1811.07819/) (2018)
- [Learning Policy Representations in Multiagent Systems](https://mlsanity.com/paper/1806.06464/) (2018)
- [A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2011.00382/) (2020)
- [A Deep Policy Inference Q-Network for Multi-Agent Systems](https://mlsanity.com/paper/1712.07893/) (2017)

## Built on by (most cited first, 1 of 1)

- [Measuring Policy Distance for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2401.11257/) (2024)

## Abstract (arXiv)

In multi-agent reinforcement learning, the inherent non-stationarity of the environment caused by other agents' actions posed significant difficulties for an agent to learn a good policy independently. One way to deal with non-stationarity is opponent modeling, by which the agent takes into consideration the influence of other agents' policies. Most existing work relies on predicting other agents' actions or goals, or discriminating between different policies. However, such modeling fails to capture the similarities and differences between policies simultaneously and thus cannot provide enough useful information when generalizing to unseen agents. To address this, we propose a general method to learn representations of other agents' policies, such that the distance between policies is deliberately reflected by the distance between representations, while the policy distance is inferred from the sampled joint action distributions during training. We empirically show that the agent conditioned on the learned policy representation can well generalize to unseen agents in three multi-agent tasks.