mlsanityData to 7 Oct 2026

Safe Exploration Using Bayesian World Models and Log-Barrier Optimization

arXiv 2405.05890 ↗First posted 9 May 2024cs.LG cs.AIcited by 4

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

A major challenge in deploying reinforcement learning in online tasks is ensuring that safety is maintained throughout the learning process. In this work, we propose CERL, a new method for solving constrained Markov decision processes while keeping the policy safe during learning. Our method leverages Bayesian world models and suggests policies that are pessimistic w.r.t. the model's epistemic uncertainty. This makes CERL robust towards model inaccuracies and leads to safe exploration during learning. In our experiments, we demonstrate that CERL outperforms the current state-of-the-art in terms of safety and optimality in solving CMDPs from image observations.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 21 of its 29 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

21 of 29 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020Mastering Atari with Discrete World Models5 Oct
  2. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  3. 2019When to Trust Your Model: Model-Based Policy Optimization19 Jun
  4. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May

Other references · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  4. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  5. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  6. 2018Safe Exploration in Continuous Action Spacescited by 237
  7. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  8. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  9. 2016Bayesian Optimization with Safety Constraints: Safe and Automatic Parameter Tuning in Roboticscited by 134
  10. 2019Linear Stochastic Bandits Under Safety Constraintscited by 90
  11. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  12. 2019Safe Exploration for Interactive Machine Learningcited by 51
5 more
  1. 2023Optimistic Active Exploration of Dynamical Systemscited by 32
  2. 2021GoSafe: Globally Optimal Safe Robot Learningcited by 27
  3. 2022GoSafeOpt: Scalable Safe Exploration for Global Optimization of Dynamical Systemscited by 17
  4. 2022Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learningcited by 11
  5. 2023Tuning Legged Locomotion Controllers via Safe Bayesian Optimizationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 22
  2. 2025Koopman-Equivariant Gaussian Processescited by 6
  3. 2026WMAttack: Automated Attack Search for Adversarial Evaluation of World-Model Agentscited by 6
  4. 2026ARB4WM: An Adversarial Robustness Benchmark for World Models in Continuous Controlcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2021Safe Continuous Control with Constrained Model-Based Policy Optimization
  2. 2026Imagine to Ensure Safety in Hierarchical Reinforcement Learning
  3. 2022Constrained Policy Optimization via Bayesian World Models
  4. 2025Safe Planning and Policy Optimization via World Model Learning
  5. 2026Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds
  6. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
  7. 2021An Offline Risk-aware Policy Selection Method for Bayesian Markov Decision Processes
  8. 2026Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning