# Robust Model-free Reinforcement Learning with Multi-objective Bayesian Optimization

arXiv: https://arxiv.org/abs/1910.13399 · HTML: https://mlsanity.com/paper/1910.13399/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2019.

## Built on by (most cited first, 16 of 16)

- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Recent Advances in Bayesian Optimization](https://mlsanity.com/paper/2206.03301/) (2022)
- [Discovering Many Diverse Solutions with Bayesian Optimization](https://mlsanity.com/paper/2210.10953/) (2022)
- [GoSafe: Globally Optimal Safe Robot Learning](https://mlsanity.com/paper/2105.13281/) (2021)
- [GoSafeOpt: Scalable Safe Exploration for Global Optimization of Dynamical Systems](https://mlsanity.com/paper/2201.09562/) (2022)
- [RAMP-Net: A Robust Adaptive MPC for Quadrotors via Physics-informed Neural Network](https://mlsanity.com/paper/2209.09025/) (2022)
- [Safe and Efficient Model-free Adaptive Control via Bayesian Optimization](https://mlsanity.com/paper/2101.07825/) (2021)
- [Minimizing UCB: a Better Local Search Strategy in Local Bayesian Optimization](https://mlsanity.com/paper/2405.15285/) (2024)
- [Towards a Theoretical Foundation of Policy Optimization for Learning Control Policies](https://mlsanity.com/paper/2210.04810/) (2022)
- [Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learning](https://mlsanity.com/paper/2203.12048/) (2022)
- [A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertainty](https://mlsanity.com/paper/2401.02914/) (2024)
- [Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation](https://mlsanity.com/paper/2112.15156/) (2021)
- [ARMOR: Robust Reinforcement Learning-based Control for UAVs under Physical Attacks](https://mlsanity.com/paper/2506.22423/) (2025)
- [Amortized Multi-Objective Optimization Across Tasks with Generative Solution Modeling](https://mlsanity.com/paper/2511.09598/) (2025)
- [Covering Multiple Objectives with a Small Set of Solutions Using Bayesian Optimization](https://mlsanity.com/paper/2501.19342/) (2025)
- [Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Control](https://mlsanity.com/paper/2510.13358/) (2025)

## Abstract (arXiv)

In reinforcement learning (RL), an autonomous agent learns to perform complex tasks by maximizing an exogenous reward signal while interacting with its environment. In real-world applications, test conditions may differ substantially from the training scenario and, therefore, focusing on pure reward maximization during training may lead to poor results at test time. In these cases, it is important to trade-off between performance and robustness while learning a policy. While several results exist for robust, model-based RL, the model-free case has not been widely investigated. In this paper, we cast the robust, model-free RL problem as a multi-objective optimization problem. To quantify the robustness of a policy, we use delay margin and gain margin, two robustness indicators that are common in control theory. We show how these metrics can be estimated from data in the model-free setting. We use multi-objective Bayesian optimization (MOBO) to solve efficiently this expensive-to-evaluate, multi-objective optimization problem. We show the benefits of our robust formulation both in sim-to-real and pure hardware experiments to balance a Furuta pendulum.