# Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models

arXiv: https://arxiv.org/abs/2102.08291 · HTML: https://mlsanity.com/paper/2102.08291/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 63 references that are in the map.
- First posted on arXiv 16 Feb 2021.

## Builds on (most cited first, 30 of 44)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Few-Shot Learning with Graph Neural Networks](https://mlsanity.com/paper/1711.04043/) (2017)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [Probabilistic Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.02817/) (2018)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [(More) Efficient Reinforcement Learning via Posterior Sampling](https://mlsanity.com/paper/1306.0940/) (2013)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Model-Ensemble Trust-Region Policy Optimization](https://mlsanity.com/paper/1802.10592/) (2018)
- [Bayesian Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.03836/) (2018)
- [Meta-Reinforcement Learning of Structured Exploration Strategies](https://mlsanity.com/paper/1802.07245/) (2018)
- [Meta-Learning with Warped Gradient Descent](https://mlsanity.com/paper/1909.00025/) (2019)
- [Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2005.06800/) (2020)
- [Exploring Model-based Planning with Policy Networks](https://mlsanity.com/paper/1906.08649/) (2019)
- [A Game Theoretic Framework for Model Based Reinforcement Learning](https://mlsanity.com/paper/2004.07804/) (2020)

## Built on by (most cited first, 17 of 17)

- [Doubly Mild Generalization for Offline Reinforcement Learning](https://mlsanity.com/paper/2411.07934/) (2024)
- [Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression](https://mlsanity.com/paper/2410.19400/) (2024)
- [A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2206.04551/) (2022)
- [MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning](https://mlsanity.com/paper/2403.09859/) (2024)
- [Practical Equivariances via Relational Conditional Neural Processes](https://mlsanity.com/paper/2306.10915/) (2023)
- [Train Hard, Fight Easy: Robust Meta Reinforcement Learning](https://mlsanity.com/paper/2301.11147/) (2023)
- [Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments](https://mlsanity.com/paper/2504.19139/) (2025)
- [Bridge the Inference Gaps of Neural Processes via Expectation Maximization](https://mlsanity.com/paper/2501.03264/) (2025)
- [Theoretical Investigations and Practical Enhancements on Tail Task Risk Minimization in Meta Learning](https://mlsanity.com/paper/2410.22788/) (2024)
- [PrivilegedDreamer: Explicit Imagination of Privileged Information for Rapid Adaptation of Learned Policies](https://mlsanity.com/paper/2502.11377/) (2025)
- [Graph Neural Networks for Relational Inductive Bias in Vision-based Deep Reinforcement Learning of Robot Control](https://mlsanity.com/paper/2203.05985/) (2022)
- [Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2511.02567/) (2025)
- [On Task-Relevant Loss Functions in Meta-Reinforcement Learning and Online LQR](https://mlsanity.com/paper/2312.05465/) (2023)
- [Amortized Network Intervention to Steer the Excitatory Point Processes](https://mlsanity.com/paper/2310.04159/) (2023)
- [VAO: Validation-Aligned Optimization for Cross-Task Generative Auto-Bidding](https://mlsanity.com/paper/2510.07760/) (2025)
- [Flow-based Policy With Distributional Reinforcement Learning in Trajectory Optimization](https://mlsanity.com/paper/2604.00977/) (2026)
- [Bridging Risk Approximation Gaps in Model Predictive Task Sampling via In-Context Modeling](https://mlsanity.com/paper/2610.14913/) (2026)

## Abstract (arXiv)

Reinforcement learning is a promising paradigm for solving sequential decision-making problems, but low data efficiency and weak generalization across tasks are bottlenecks in real-world applications. Model-based meta reinforcement learning addresses these issues by learning dynamics and leveraging knowledge from prior experience. In this paper, we take a closer look at this framework, and propose a new Thompson-sampling based approach that consists of a new model to identify task dynamics together with an amortized policy optimization step. We show that our model, called a graph structured surrogate model (GSSM), outperforms state-of-the-art methods in predicting environment dynamics. Additionally, our approach is able to obtain high returns, while allowing fast execution during deployment by avoiding test time policy gradient optimization.