# Agent Modelling under Partial Observability for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2006.09447 · HTML: https://mlsanity.com/paper/2006.09447/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 30 of 57)

- [MADiff: Offline Multi-agent Learning with Diffusion Models](https://mlsanity.com/paper/2305.17330/) (2023)
- [Learning to Cooperate with Humans using Generative Agents](https://mlsanity.com/paper/2411.13934/) (2024)
- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversity](https://mlsanity.com/paper/2207.14138/) (2022)
- [N-Agent Ad Hoc Teamwork](https://mlsanity.com/paper/2404.10740/) (2024)
- [Minimum Coverage Sets for Training Robust Ad Hoc Teamwork Agents](https://mlsanity.com/paper/2308.09595/) (2023)
- [A General Learning Framework for Open Ad Hoc Teamwork Using Graph-based Policy Learning](https://mlsanity.com/paper/2210.05448/) (2022)
- [Interactive Autonomous Navigation with Internal State Inference and Interactivity Estimation](https://mlsanity.com/paper/2311.16091/) (2023)
- [Fast Peer Adaptation with Context-aware Exploration](https://mlsanity.com/paper/2402.02468/) (2024)
- [SUB-PLAY: Adversarial Policies against Partially Observed Multi-Agent Reinforcement Learning Systems](https://mlsanity.com/paper/2402.03741/) (2024)
- [Verified Probabilistic Policies for Deep Reinforcement Learning](https://mlsanity.com/paper/2201.03698/) (2022)
- [Multi-Agent Inverse Reinforcement Learning in Real World Unstructured Pedestrian Crowds](https://mlsanity.com/paper/2405.16439/) (2024)
- [ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork](https://mlsanity.com/paper/2505.23686/) (2025)
- [Learning Task Embeddings for Teamwork Adaptation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2207.02249/) (2022)
- [iPLAN: Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.06236/) (2023)
- [Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2410.17373/) (2024)
- [TacoMAS: Test-Time Co-Evolution of Topology and Capability in LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2605.09539/) (2026)
- [Self-Confirming Transformer for Belief-Conditioned Adaptation in Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2310.04579/) (2023)
- [Partner Modelling Emerges in Recurrent Agents (But Only When It Matters)](https://mlsanity.com/paper/2505.17323/) (2025)
- [Deep Reinforcement Learning for Multi-Agent Interaction](https://mlsanity.com/paper/2208.01769/) (2022)
- [Centralized Training with Hybrid Execution in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.06274/) (2022)
- [CAMMARL: Conformal Action Modeling in Multi Agent Reinforcement Learning](https://mlsanity.com/paper/2306.11128/) (2023)
- [Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration](https://mlsanity.com/paper/2505.05262/) (2025)
- [Multi-Agent Systems Should be Treated as Principal-Agent Problems](https://mlsanity.com/paper/2601.23211/) (2026)
- [Decision-making with Speculative Opponent Models](https://mlsanity.com/paper/2211.11940/) (2022)
- [Selective Reincarnation: Offline-to-Online Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.00977/) (2023)
- [Contrastive learning-based agent modeling for deep reinforcement learning](https://mlsanity.com/paper/2401.00132/) (2023)
- [Trajectory-Class-Aware Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.01440/) (2025)
- [EMOTE: An Explainable architecture for Modelling the Other Through Empathy](https://mlsanity.com/paper/2306.00295/) (2023)
- [Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learning](https://mlsanity.com/paper/2307.05209/) (2023)

## Abstract (arXiv)

Modelling the behaviours of other agents is essential for understanding how agents interact and making effective decisions. Existing methods for agent modelling commonly assume knowledge of the local observations and chosen actions of the modelled agents during execution. To eliminate this assumption, we extract representations from the local information of the controlled agent using encoder-decoder architectures. Using the observations and actions of the modelled agents during training, our models learn to extract representations about the modelled agents conditioned only on the local observations of the controlled agent. The representations are used to augment the controlled agent's decision policy which is trained via deep reinforcement learning; thus, during execution, the policy does not require access to other agents' information. We provide a comprehensive evaluation and ablations studies in cooperative, competitive and mixed multi-agent environments, showing that our method achieves higher returns than baseline methods which do not use the learned representations.