# Agent Modeling as Auxiliary Task for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1907.09597 · HTML: https://mlsanity.com/paper/1907.09597/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2019.

## Built on by (most cited first, 14 of 14)

- [End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes](https://mlsanity.com/paper/2305.15930/) (2023)
- [Deep Interactive Bayesian Reinforcement Learning via Meta-Learning](https://mlsanity.com/paper/2101.03864/) (2021)
- [PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation](https://mlsanity.com/paper/2109.11094/) (2021)
- [PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning](https://mlsanity.com/paper/2102.12560/) (2021)
- [ELIGN: Expectation Alignment as a Multi-Agent Intrinsic Reward](https://mlsanity.com/paper/2210.04365/) (2022)
- [Interactive Autonomous Navigation with Internal State Inference and Interactivity Estimation](https://mlsanity.com/paper/2311.16091/) (2023)
- [The Curse of Diversity in Ensemble-Based Exploration](https://mlsanity.com/paper/2405.04342/) (2024)
- [Decentralized Multi-Agent Reinforcement Learning with Global State Prediction](https://mlsanity.com/paper/2306.12926/) (2023)
- [Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration](https://mlsanity.com/paper/2505.05262/) (2025)
- [BRExIt: On Opponent Modelling in Expert Iteration](https://mlsanity.com/paper/2206.00113/) (2022)
- [A novel multi-agent dynamic portfolio optimization learning system based on hierarchical deep reinforcement learning](https://mlsanity.com/paper/2501.06832/) (2025)
- [Fact-based Agent modeling for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2310.12290/) (2023)
- [Reward Shaping for Improved Learning in Real-time Strategy Game Play](https://mlsanity.com/paper/2311.16339/) (2023)
- [A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigation](https://mlsanity.com/paper/2607.09792/) (2026)

## Abstract (arXiv)

In this paper we explore how actor-critic methods in deep reinforcement learning, in particular Asynchronous Advantage Actor-Critic (A3C), can be extended with agent modeling. Inspired by recent works on representation learning and multiagent deep reinforcement learning, we propose two architectures to perform agent modeling: the first one based on parameter sharing, and the second one based on agent policy features. Both architectures aim to learn other agents' policies as auxiliary tasks, besides the standard actor (policy) and critic (values). We performed experiments in both cooperative and competitive domains. The former is a problem of coordinated multiagent object transportation and the latter is a two-player mini version of the Pommerman game. Our results show that the proposed architectures stabilize learning and outperform the standard A3C architecture when learning a best response in terms of expected rewards.