# SafeAPT: Safe Simulation-to-Real Robot Learning using Diverse Policies Learned in Simulation

arXiv: https://arxiv.org/abs/2201.13248 · HTML: https://mlsanity.com/paper/2201.13248/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 27 Jan 2022.

## Builds on (most cited first, 18 of 18)

- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems](https://mlsanity.com/paper/1705.01292/) (2017)
- [Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping](https://mlsanity.com/paper/1709.07857/) (2017)
- [Bayesian Optimization with Safety Constraints: Safe and Automatic Parameter Tuning in Robotics](https://mlsanity.com/paper/1602.04450/) (2016)
- [Quality and Diversity Optimization: A Unifying Modular Framework](https://mlsanity.com/paper/1708.09251/) (2017)
- [Conservative Safety Critics for Exploration](https://mlsanity.com/paper/2010.14497/) (2020)
- [Cautious Model Predictive Control using Gaussian Process Regression](https://mlsanity.com/paper/1705.10702/) (2017)
- [Quality-Diversity Optimization: a novel branch of stochastic optimization](https://mlsanity.com/paper/2012.04322/) (2020)
- [Meta Reinforcement Learning for Sim-to-real Domain Adaptation](https://mlsanity.com/paper/1909.12906/) (2019)
- [Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings](https://mlsanity.com/paper/2008.06622/) (2020)
- [Black-Box Data-efficient Policy Search for Robotics](https://mlsanity.com/paper/1703.07261/) (2017)
- [Adaptive Prior Selection for Repertoire-based Online Adaptation in Robotics](https://mlsanity.com/paper/1907.07029/) (2019)
- [Multi-objective Model-based Policy Search for Data-efficient Learning with Sparse Rewards](https://mlsanity.com/paper/1806.09351/) (2018)
- [Few-shot model-based adaptation in noisy conditions](https://mlsanity.com/paper/2010.08397/) (2020)
- [Safety-Aware Robot Damage Recovery Using Constrained Bayesian Optimization and Simulated Priors](https://mlsanity.com/paper/1611.09419/) (2016)

## Built on by (most cited first, 10 of 10)

- [Safe Reinforcement Learning of Dynamic High-Dimensional Robotic Tasks: Navigation, Manipulation, Interaction](https://mlsanity.com/paper/2209.13308/) (2022)
- [A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviors](https://mlsanity.com/paper/2302.13137/) (2023)
- [SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer](https://mlsanity.com/paper/2509.18648/) (2025)
- [Embodied AI: Emerging Risks and Opportunities for Policy Action](https://mlsanity.com/paper/2509.00117/) (2025)
- [Sim-and-Real Reinforcement Learning for Manipulation: A Consensus-based Approach](https://mlsanity.com/paper/2302.13423/) (2023)
- [Machine Learning Meets Advanced Robotic Manipulation](https://mlsanity.com/paper/2309.12560/) (2023)
- [CURE: Simulation-Augmented Auto-Tuning in Robotics](https://mlsanity.com/paper/2402.05399/) (2024)
- [Open-Ended Diverse Solution Discovery with Regulated Behavior Patterns for Cross-Domain Adaptation](https://mlsanity.com/paper/2209.12029/) (2022)
- [Sim-to-Real Deep Reinforcement Learning with Manipulators for Pick-and-place](https://mlsanity.com/paper/2309.09247/) (2023)
- [Towards provable probabilistic safety for scalable embodied AI systems](https://mlsanity.com/paper/2506.05171/) (2025)

## Abstract (arXiv)

The framework of Simulation-to-real learning, i.e, learning policies in simulation and transferring those policies to the real world is one of the most promising approaches towards data-efficient learning in robotics. However, due to the inevitable reality gap between the simulation and the real world, a policy learned in the simulation may not always generate a safe behaviour on the real robot. As a result, during adaptation of the policy in the real world, the robot may damage itself or cause harm to its surroundings. In this work, we introduce a novel learning algorithm called SafeAPT that leverages a diverse repertoire of policies evolved in the simulation and transfers the most promising safe policy to the real robot through episodic interaction. To achieve this, SafeAPT iteratively learns a probabilistic reward model as well as a safety model using real-world observations combined with simulated experiences as priors. Then, it performs Bayesian optimization on the repertoire with the reward model while maintaining the specified safety constraint using the safety model. SafeAPT allows a robot to adapt to a wide range of goals safely with the same repertoire of policies evolved in the simulation. We compare SafeAPT with several baselines, both in simulated and real robotic experiments and show that SafeAPT finds high-performance policies within a few minutes in the real world while minimizing safety violations during the interactions.