# Improved Robustness and Safety for Pre-Adaptation of Meta Reinforcement Learning with Prior Regularization

arXiv: https://arxiv.org/abs/2108.08448 · HTML: https://mlsanity.com/paper/2108.08448/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 19 Aug 2021.

## Builds on (most cited first, 21 of 21)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Adversarial Attacks on Graph Neural Networks via Meta Learning](https://mlsanity.com/paper/1902.08412/) (2019)
- [Meta-Reinforcement Learning of Structured Exploration Strategies](https://mlsanity.com/paper/1802.07245/) (2018)
- [Improving Generalization in Meta Reinforcement Learning using Learned Objectives](https://mlsanity.com/paper/1910.04098/) (2019)
- [Weakly-Convex Concave Min-Max Optimization: Provable Algorithms and Applications in Machine Learning](https://mlsanity.com/paper/1810.02060/) (2018)
- [Meta-Q-Learning](https://mlsanity.com/paper/1910.00125/) (2019)
- [Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimization](https://mlsanity.com/paper/2003.01303/) (2020)
- [Model-based Adversarial Meta-Reinforcement Learning](https://mlsanity.com/paper/2006.08875/) (2020)
- [Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloads](https://mlsanity.com/paper/2004.11345/) (2020)
- [Guided Meta-Policy Search](https://mlsanity.com/paper/1904.00956/) (2019)
- [Online Meta-Critic Learning for Off-Policy Actor-Critic Methods](https://mlsanity.com/paper/2003.05334/) (2020)
- [Meta-Adversarial Inverse Reinforcement Learning for Decision-making Tasks](https://mlsanity.com/paper/2103.12694/) (2021)
- [Composing Meta-Policies for Autonomous Driving Using Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/1711.01503/) (2017)
- [Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning](https://mlsanity.com/paper/2104.08876/) (2021)

## Built on by (most cited first, 4 of 4)

- [Safe Continual Reinforcement Learning Methods for Nonstationary Environments. Towards a Survey of the State of the Art](https://mlsanity.com/paper/2601.05152/) (2026)
- [TLXML: Task-Level Explanation of Meta-Learning via Influence Functions](https://mlsanity.com/paper/2501.14271/) (2025)
- [Adjustment Speed as a Safety Constraint for Nonstationary Reinforcement Learning](https://mlsanity.com/paper/2607.21646/) (2026)
- [Proactive Context-Forecasted Safety Constraints for Nonstationary Reinforcement Learning](https://mlsanity.com/paper/2609.08080/) (2026)

## Abstract (arXiv)

Meta Reinforcement Learning (Meta-RL) has seen substantial advancements recently. In particular, off-policy methods were developed to improve the data efficiency of Meta-RL techniques. textitProbabilistic embeddings for actor-critic RL (PEARL) is a leading approach for multi-MDP adaptation problems. A major drawback of many existing Meta-RL methods, including PEARL, is that they do not explicitly consider the safety of the prior policy when it is exposed to a new task for the first time. Safety is essential for many real-world applications, including field robots and Autonomous Vehicles (AVs). In this paper, we develop the PEARL PLUS (PEARL^+) algorithm, which optimizes the policy for both prior (pre-adaptation) safety and posterior (after-adaptation) performance. Building on top of PEARL, our proposed PEARL^+ algorithm introduces a prior regularization term in the reward function and a new Q-network for recovering the state-action value under prior context assumptions, to improve the robustness to task distribution shift and safety of the trained network exposed to a new task for the first time. The performance of PEARL^+ is validated by solving three safety-critical problems related to robots and AVs, including two MuJoCo benchmark problems. From the simulation experiments, we show that safety of the prior policy is significantly improved and more robust to task distribution shift compared to PEARL.