# Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials

arXiv: https://arxiv.org/abs/2002.03072 · HTML: https://mlsanity.com/paper/2002.03072/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2020.

## Built on by (most cited first, 20 of 20)

- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.05265/) (2021)
- [AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents](https://mlsanity.com/paper/2310.09971/) (2023)
- [Factored Adaptation for Non-Stationary Reinforcement Learning](https://mlsanity.com/paper/2203.16582/) (2022)
- [Block Contextual MDPs for Continual Learning](https://mlsanity.com/paper/2110.06972/) (2021)
- [Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalization](https://mlsanity.com/paper/2403.10967/) (2024)
- [MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning](https://mlsanity.com/paper/2403.09859/) (2024)
- [Toward AI Assistants That Let Designers Design](https://mlsanity.com/paper/2107.13074/) (2021)
- [Data-Efficient Task Generalization via Probabilistic Model-based Meta Reinforcement Learning](https://mlsanity.com/paper/2311.07558/) (2023)
- [Zero-Shot Assistance in Sequential Decision Problems](https://mlsanity.com/paper/2202.07364/) (2022)
- [A model-based approach to meta-Reinforcement Learning: Transformers and tree search](https://mlsanity.com/paper/2208.11535/) (2022)
- [Offline Reinforcement Learning with Domain-Unlabeled Data](https://mlsanity.com/paper/2404.07465/) (2024)
- [GalilAI: Out-of-Task Distribution Detection using Causal Active Experimentation for Safe Transfer RL](https://mlsanity.com/paper/2110.15489/) (2021)
- [Policy Resilience to Environment Poisoning Attacks on Reinforcement Learning](https://mlsanity.com/paper/2304.12151/) (2023)
- [Heterogeneous Knowledge for Augmented Modular Reinforcement Learning](https://mlsanity.com/paper/2306.01158/) (2023)
- [Situationally-Aware Dynamics Learning](https://mlsanity.com/paper/2505.19574/) (2025)
- [Contextual Intelligence The Next Leap for Reinforcement Learning](https://mlsanity.com/paper/2604.02348/) (2026)
- [Compositional Q-learning for electrolyte repletion with imbalanced patient sub-populations](https://mlsanity.com/paper/2110.02879/) (2021)
- [On the Benefits of Leveraging Structural Information in Planning Over the Learned Model](https://mlsanity.com/paper/2303.08856/) (2023)
- [Meta-RL with Bayesian Linear Task Models](https://mlsanity.com/paper/2512.20974/) (2025)

## Abstract (arXiv)

There is broad interest in creating RL agents that can solve many (related) tasks and adapt to new tasks and environments after initial training. Model-based RL leverages learned surrogate models that describe dynamics and rewards of individual tasks, such that planning in a good surrogate can lead to good control of the true system. Rather than solving each task individually from scratch, hierarchical models can exploit the fact that tasks are often related by (unobserved) causal factors of variation in order to achieve efficient generalization, as in learning how the mass of an item affects the force required to lift it can generalize to previously unobserved masses. We propose Generalized Hidden Parameter MDPs (GHP-MDPs) that describe a family of MDPs where both dynamics and reward can change as a function of hidden parameters that vary across tasks. The GHP-MDP augments model-based RL with latent variables that capture these hidden parameters, facilitating transfer across tasks. We also explore a variant of the model that incorporates explicit latent structure mirroring the causal factors of variation across tasks (for instance: agent properties, environmental factors, and goals). We experimentally demonstrate state-of-the-art performance and sample-efficiency on a new challenging MuJoCo task using reward and dynamics latent spaces, while beating a previous state-of-the-art baseline with >10× less data. Using test-time inference of the latent variables, our approach generalizes in a single episode to novel combinations of dynamics and reward, and to novel rewards.