# Deep Interactive Bayesian Reinforcement Learning via Meta-Learning

arXiv: https://arxiv.org/abs/2101.03864 · HTML: https://mlsanity.com/paper/2101.03864/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 56 references that are in the map.
- First posted on arXiv 11 Jan 2021.

## Builds on (most cited first, 26 of 26)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Actor-Attention-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1810.02912/) (2018)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems](https://mlsanity.com/paper/1709.08071/) (2017)
- [Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1807.09936/) (2018)
- [Opponent Modeling in Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05559/) (2016)
- [A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity](https://mlsanity.com/paper/1707.09183/) (2017)
- [Meta reinforcement learning as task inference](https://mlsanity.com/paper/1905.06424/) (2019)
- [Meta-learning of Sequential Strategies](https://mlsanity.com/paper/1905.03030/) (2019)
- [Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1811.01458/) (2018)
- [Learning Policy Representations in Multiagent Systems](https://mlsanity.com/paper/1806.06464/) (2018)
- [Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1912.02288/) (2019)
- [Finding Friend and Foe in Multi-Agent Games](https://mlsanity.com/paper/1906.02330/) (2019)
- [Reasoning about Hypothetical Agent Behaviours and their Parameters](https://mlsanity.com/paper/1906.11064/) (2019)
- [Belief and Truth in Hypothesised Behaviours](https://mlsanity.com/paper/1507.07688/) (2015)
- [Variational Autoencoders for Opponent Modeling in Multi-Agent Systems](https://mlsanity.com/paper/2001.10829/) (2020)
- [Agent Modeling as Auxiliary Task for Deep Reinforcement Learning](https://mlsanity.com/paper/1907.09597/) (2019)
- [Learning to Play against Any Mixture of Opponents](https://mlsanity.com/paper/2009.14180/) (2020)
- [Cooperative Games with Overlapping Coalitions](https://mlsanity.com/paper/1401.3856/) (2014)
- [Monte Carlo Planning method estimates planning horizons during interactive social exchange](https://mlsanity.com/paper/1502.03696/) (2015)
- [Evaluating the Rainbow DQN Agent in Hanabi with Unseen Partners](https://mlsanity.com/paper/2004.13291/) (2020)

## Built on by (most cited first, 29 of 29)

- [A Survey of Ad Hoc Teamwork Research](https://mlsanity.com/paper/2202.10450/) (2022)
- [Learning to Ground Multi-Agent Communication with Autoencoders](https://mlsanity.com/paper/2110.15349/) (2021)
- [Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversity](https://mlsanity.com/paper/2207.14138/) (2022)
- [N-Agent Ad Hoc Teamwork](https://mlsanity.com/paper/2404.10740/) (2024)
- [Minimum Coverage Sets for Training Robust Ad Hoc Teamwork Agents](https://mlsanity.com/paper/2308.09595/) (2023)
- [Interactive Autonomous Navigation with Internal State Inference and Interactivity Estimation](https://mlsanity.com/paper/2311.16091/) (2023)
- [Fast Peer Adaptation with Context-aware Exploration](https://mlsanity.com/paper/2402.02468/) (2024)
- [Open Ad Hoc Teamwork with Cooperative Game Theory](https://mlsanity.com/paper/2402.15259/) (2024)
- [ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork](https://mlsanity.com/paper/2505.23686/) (2025)
- [Generalized Beliefs for Cooperative AI](https://mlsanity.com/paper/2206.12765/) (2022)
- [Learning Task Embeddings for Teamwork Adaptation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2207.02249/) (2022)
- [Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformers](https://mlsanity.com/paper/2504.04395/) (2025)
- [Partner Modelling Emerges in Recurrent Agents (But Only When It Matters)](https://mlsanity.com/paper/2505.17323/) (2025)
- [Shapley Machine: A Game-Theoretic Framework for N-Agent Ad Hoc Teamwork](https://mlsanity.com/paper/2506.11285/) (2025)
- [Adaptively Coordinating with Novel Partners via Learned Latent Strategies](https://mlsanity.com/paper/2511.12754/) (2025)
- [CAMMARL: Conformal Action Modeling in Multi Agent Reinforcement Learning](https://mlsanity.com/paper/2306.11128/) (2023)
- [Inverse Attention Agents for Multi-Agent Systems](https://mlsanity.com/paper/2410.21794/) (2024)
- [Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration](https://mlsanity.com/paper/2505.05262/) (2025)
- [Decision-making with Speculative Opponent Models](https://mlsanity.com/paper/2211.11940/) (2022)
- [Knowledge-based Reasoning and Learning under Partial Observability in Ad Hoc Teamwork](https://mlsanity.com/paper/2306.00790/) (2023)
- [Modeling Latent Partner Strategies for Adaptive Zero-Shot Human-Agent Collaboration](https://mlsanity.com/paper/2507.05244/) (2025)
- [JaxAHT: A JAX-Based Library for Ad Hoc Teamwork](https://mlsanity.com/paper/2609.13716/) (2026)
- [Learning to Infer Belief Embedded Communication](https://mlsanity.com/paper/2203.07832/) (2022)
- [Knowledge-based and Data-driven Reasoning and Learning for Ad Hoc Teamwork](https://mlsanity.com/paper/2208.11556/) (2022)
- [Policy Dispersion in Non-Markovian Environment](https://mlsanity.com/paper/2302.14509/) (2023)
- [An Advanced Reinforcement Learning Framework for Online Scheduling of Deferrable Workloads in Cloud Computing](https://mlsanity.com/paper/2406.01047/) (2024)
- [Generalizable Agent Modeling for Agent Collaboration-Competition Adaptation with Multi-Retrieval and Dynamic Generation](https://mlsanity.com/paper/2506.16718/) (2025)
- [Remembering the Markov Property in Cooperative MARL](https://mlsanity.com/paper/2507.18333/) (2025)
- [Generalized Intention Modeling in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.31318/) (2026)

## Abstract (arXiv)

Agents that interact with other agents often do not know a priori what the other agents' strategies are, but have to maximise their own online return while interacting with and learning about others. The optimal adaptive behaviour under uncertainty over the other agents' strategies w.r.t. some prior can in principle be computed using the Interactive Bayesian Reinforcement Learning framework. Unfortunately, doing so is intractable in most settings, and existing approximation methods are restricted to small tasks. To overcome this, we propose to meta-learn approximate belief inference and Bayes-optimal behaviour for a given prior. To model beliefs over other agents, we combine sequential and hierarchical Variational Auto-Encoders, and meta-train this inference model alongside the policy. We show empirically that our approach outperforms existing methods that use a model-free approach, sample from the approximate posterior, maintain memory-free models of others, or do not fully utilise the known structure of the environment.