# VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning

arXiv: https://arxiv.org/abs/1910.08348 · HTML: https://mlsanity.com/paper/1910.08348/

## Key facts

- Cited by 217 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Oct 2019.

## Built on by (most cited first, 30 of 217)

- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Supervised Pretraining Can Learn In-Context Reinforcement Learning](https://mlsanity.com/paper/2306.14892/) (2023)
- [Human-Timescale Adaptation in an Open-Ended Task Space](https://mlsanity.com/paper/2301.07608/) (2023)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2305.18459/) (2023)
- [Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability](https://mlsanity.com/paper/2107.06277/) (2021)
- [Offline Meta-Reinforcement Learning with Advantage Weighting](https://mlsanity.com/paper/2008.06043/) (2020)
- [Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining](https://mlsanity.com/paper/2310.08566/) (2023)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX](https://mlsanity.com/paper/2312.12044/) (2023)
- [Offline Meta-Reinforcement Learning with Online Self-Supervision](https://mlsanity.com/paper/2107.03974/) (2021)
- [Offline RL Policies Should be Trained to be Adaptive](https://mlsanity.com/paper/2207.02200/) (2022)
- [Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment](https://mlsanity.com/paper/2104.05632/) (2021)
- [Transformers are Meta-Reinforcement Learners](https://mlsanity.com/paper/2206.06614/) (2022)
- [Skill-based Meta-Reinforcement Learning](https://mlsanity.com/paper/2204.11828/) (2022)
- [Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents](https://mlsanity.com/paper/2102.02926/) (2021)
- [Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning](https://mlsanity.com/paper/2206.10442/) (2022)
- [Choreographer: Learning and Adapting Skills in Imagination](https://mlsanity.com/paper/2211.13350/) (2022)
- [Cross-Domain Policy Adaptation via Value-Guided Data Filtering](https://mlsanity.com/paper/2305.17625/) (2023)
- [Deep Interactive Bayesian Reinforcement Learning via Meta-Learning](https://mlsanity.com/paper/2101.03864/) (2021)
- [Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learning](https://mlsanity.com/paper/2207.04703/) (2022)
- [You Only Live Once: Single-Life Reinforcement Learning](https://mlsanity.com/paper/2210.08863/) (2022)
- [State Regularized Policy Optimization on Data with Dynamics Shift](https://mlsanity.com/paper/2306.03552/) (2023)
- [Maximum State Entropy Exploration using Predecessor and Successor Representations](https://mlsanity.com/paper/2306.14808/) (2023)
- [Learning Options via Compression](https://mlsanity.com/paper/2212.04590/) (2022)
- [Accelerating Exploration with Unlabeled Prior Data](https://mlsanity.com/paper/2311.05067/) (2023)
- [Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policies](https://mlsanity.com/paper/2310.16686/) (2023)
- [Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions](https://mlsanity.com/paper/2402.13777/) (2024)

## Abstract (arXiv)

Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning. A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment. Computing a Bayes-optimal policy is however intractable for all but the smallest tasks. In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncertainty directly during action selection. In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty. We further evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher online return than existing methods.