# Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings

arXiv: https://arxiv.org/abs/2008.06622 · HTML: https://mlsanity.com/paper/2008.06622/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2020.

## Built on by (most cited first, 30 of 42)

- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [A Survey on Model-based Reinforcement Learning](https://mlsanity.com/paper/2206.09328/) (2022)
- [Learning to Synthesize Programs as Interpretable and Generalizable Policies](https://mlsanity.com/paper/2108.13643/) (2021)
- [How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review](https://mlsanity.com/paper/2107.12045/) (2021)
- [Safe Driving via Expert Guided Policy Optimization](https://mlsanity.com/paper/2110.06831/) (2021)
- [MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation](https://mlsanity.com/paper/2309.14236/) (2023)
- [One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2212.00124/) (2022)
- [Robust Policy Learning over Multiple Uncertainty Sets](https://mlsanity.com/paper/2202.07013/) (2022)
- [Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learning](https://mlsanity.com/paper/2112.07701/) (2021)
- [Safe Exploration by Solving Early Terminated MDP](https://mlsanity.com/paper/2107.04200/) (2021)
- [Distributionally Robust Constrained Reinforcement Learning under Strong Duality](https://mlsanity.com/paper/2406.15788/) (2024)
- [Robust Fast Adaptation from Adversarially Explicit Task Distribution Generation](https://mlsanity.com/paper/2407.19523/) (2024)
- [Lyapunov Barrier Policy Optimization](https://mlsanity.com/paper/2103.09230/) (2021)
- [Probabilistic Constraint for Safety-Critical Reinforcement Learning](https://mlsanity.com/paper/2306.17279/) (2023)
- [LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks](https://mlsanity.com/paper/2107.04775/) (2021)
- [SafeAPT: Safe Simulation-to-Real Robot Learning using Diverse Policies Learned in Simulation](https://mlsanity.com/paper/2201.13248/) (2022)
- [A Simple Yet Effective Strategy to Robustify the Meta Learning Paradigm](https://mlsanity.com/paper/2310.00708/) (2023)
- [Monitored Markov Decision Processes](https://mlsanity.com/paper/2402.06819/) (2024)
- [Learning Differentiable Safety-Critical Control using Control Barrier Functions for Generalization to Novel Environments](https://mlsanity.com/paper/2201.01347/) (2022)
- [MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance](https://mlsanity.com/paper/2112.03575/) (2021)
- [Hybrid Action Based Reinforcement Learning for Multi-Objective Compatible Autonomous Driving](https://mlsanity.com/paper/2501.08096/) (2025)
- [Reinforcement Learning by Guided Safe Exploration](https://mlsanity.com/paper/2307.14316/) (2023)
- [Effects of Safety State Augmentation on Safe Exploration](https://mlsanity.com/paper/2206.02675/) (2022)
- [Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Intervention](https://mlsanity.com/paper/2111.05819/) (2021)
- [Laplacian Representations for Decision-Time Planning](https://mlsanity.com/paper/2602.05031/) (2026)
- [Learning to Be Cautious](https://mlsanity.com/paper/2110.15907/) (2021)
- [Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planning](https://mlsanity.com/paper/2111.04972/) (2021)
- [Safe Reinforcement Learning with Contrastive Risk Prediction](https://mlsanity.com/paper/2209.09648/) (2022)
- [Approximate Shielding of Atari Agents for Safe Exploration](https://mlsanity.com/paper/2304.11104/) (2023)
- [Model-Assisted Probabilistic Safe Adaptive Control With Meta-Bayesian Learning](https://mlsanity.com/paper/2307.00828/) (2023)

## Abstract (arXiv)

Reinforcement learning (RL) in real-world safety-critical target settings like urban driving is hazardous, imperiling the RL agent, other agents, and the environment. To overcome this difficulty, we propose a "safety-critical adaptation" task setting: an agent first trains in non-safety-critical "source" environments such as in a simulator, before it adapts to the target environment where failures carry heavy costs. We propose a solution approach, CARL, that builds on the intuition that prior experience in diverse environments equips an agent to estimate risk, which in turn enables relative safety through risk-averse, cautious adaptation. CARL first employs model-based RL to train a probabilistic model to capture uncertainty about transition dynamics and catastrophic states across varied source environments. Then, when exploring a new safety-critical environment with unknown dynamics, the CARL agent plans to avoid actions that could lead to catastrophic states. In experiments on car driving, cartpole balancing, half-cheetah locomotion, and robotic object manipulation, CARL successfully acquires cautious exploration behaviors, yielding higher rewards with fewer failures than strong RL adaptation baselines. Website at https://sites.google.com/berkeley.edu/carl.