# Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling

arXiv: https://arxiv.org/abs/2006.07178 · HTML: https://mlsanity.com/paper/2006.07178/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2020.

## Built on by (most cited first, 26 of 26)

- [Test-Time Adaptation via Self-Training with Nearest Neighbor Information](https://mlsanity.com/paper/2207.10792/) (2022)
- [On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning](https://mlsanity.com/paper/2206.03271/) (2022)
- [Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models](https://mlsanity.com/paper/2102.08291/) (2021)
- [Causal Multi-Agent Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2111.06721/) (2021)
- [Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture](https://mlsanity.com/paper/2105.13524/) (2021)
- [Train Hard, Fight Easy: Robust Meta Reinforcement Learning](https://mlsanity.com/paper/2301.11147/) (2023)
- [Offline Meta Reinforcement Learning with In-Distribution Online Adaptation](https://mlsanity.com/paper/2305.19529/) (2023)
- [Distributionally Adaptive Meta Reinforcement Learning](https://mlsanity.com/paper/2210.03104/) (2022)
- [Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learning](https://mlsanity.com/paper/2305.00286/) (2023)
- [Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations](https://mlsanity.com/paper/2312.15909/) (2023)
- [Linear Representation Meta-Reinforcement Learning for Instant Adaptation](https://mlsanity.com/paper/2101.04750/) (2021)
- [Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deployment](https://mlsanity.com/paper/2311.01059/) (2023)
- [On the Practical Consistency of Meta-Reinforcement Learning Algorithms](https://mlsanity.com/paper/2112.00478/) (2021)
- [Dream to Adapt: Meta Reinforcement Learning by Latent Context Imagination and MDP Imagination](https://mlsanity.com/paper/2311.06673/) (2023)
- [Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learning](https://mlsanity.com/paper/2502.03752/) (2025)
- [Hindsight Foresight Relabeling for Meta-Reinforcement Learning](https://mlsanity.com/paper/2109.09031/) (2021)
- [A model-based approach to meta-Reinforcement Learning: Transformers and tree search](https://mlsanity.com/paper/2208.11535/) (2022)
- [Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Tasks](https://mlsanity.com/paper/2502.02834/) (2025)
- [A Meta-Learning Approach for Multi-Objective Reinforcement Learning in Sustainable Home Environments](https://mlsanity.com/paper/2407.11489/) (2024)
- [Adaptive Adversarial Training for Meta Reinforcement Learning](https://mlsanity.com/paper/2104.13302/) (2021)
- [Fast Model-based Policy Search for Universal Policy Networks](https://mlsanity.com/paper/2202.05843/) (2022)
- [In-Context Reinforcement Learning under Non-Stationarity: A Survey](https://mlsanity.com/paper/2607.11906/) (2026)
- [Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and Environments](https://mlsanity.com/paper/2101.07599/) (2021)
- [DIRECT: Learning from Sparse and Shifting Rewards using Discriminative Reward Co-Training](https://mlsanity.com/paper/2301.07421/) (2023)
- [Label Distribution Shift-Aware Prediction Refinement for Test-Time Adaptation](https://mlsanity.com/paper/2411.15204/) (2024)
- [Offline-Online Reinforcement Learning for Linear Mixture MDPs](https://mlsanity.com/paper/2604.11994/) (2026)

## Abstract (arXiv)

Reinforcement learning algorithms can acquire policies for complex tasks autonomously. However, the number of samples required to learn a diverse set of skills can be prohibitively large. While meta-reinforcement learning methods have enabled agents to leverage prior experience to adapt quickly to new tasks, their performance depends crucially on how close the new task is to the previously experienced tasks. Current approaches are either not able to extrapolate well, or can do so at the expense of requiring extremely large amounts of data for on-policy meta-training. In this work, we present model identification and experience relabeling (MIER), a meta-reinforcement learning algorithm that is both efficient and extrapolates well when faced with out-of-distribution tasks at test time. Our method is based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data, more easily than policies and value functions. These dynamics models can then be used to continue training policies and value functions for out-of-distribution tasks without using meta-reinforcement learning at all, by generating synthetic experience for the new task.