Data to 7 Oct 2026

Constrained Policy Optimization via Bayesian World Models

arXiv 2201.09802 ↗First posted 24 Jan 2022cs.LG cs.AI cs.ROcited by 55

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Improving sample-efficiency and safety are crucial challenges when deploying reinforcement learning in high-stakes real world applications. We propose LAMBDA, a novel model-based approach for policy optimization in safety critical tasks modeled via constrained Markov decision processes. Our approach utilizes Bayesian world models, and harnesses the resulting uncertainty to maximize optimistic upper bounds on the task objective, as well as pessimistic upper bounds on the safety constraints. We demonstrate LAMBDA's state of the art performance on the Safety-Gym benchmark suite in terms of sample efficiency and constraint violation.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 30 of its 52 references that are in the map. First posted on arXiv 24 Jan 2022.

Builds on

30 of 52 references.

Closest ancestors · 3

Same direction, newest first

Other references · 27

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2017Proximal Policy Optimization Algorithmscited by 14,441
2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
2016Concrete Problems in AI Safetycited by 1,582
2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302

15 more

2019Monte Carlo Gradient Estimation in Machine Learningcited by 281
2018Safe Exploration in Continuous Action Spacescited by 237
2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
2020Exploration-Exploitation in Constrained MDPscited by 146
2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
2018Learning-based Model Predictive Control for Safe Explorationcited by 116
2020Conservative Safety Critics for Explorationcited by 107
2018A predictive safety filter for learning-based control of constrained nonlinear dynamical systemscited by 91
2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
2020Safe Reinforcement Learning via Curriculum Inductioncited by 73
2019A Bayesian Approach to Robust Reinforcement Learningcited by 32
2021Safe Continuous Control with Constrained Model-Based Policy Optimizationcited by 17
2013Scaling Up Robust MDPs by Reinforcement Learningcited by 16
2021Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 17; 2024: 11; 2025: 9; 2026: 9.

55 papers.

Sort:

By area:

2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 54
2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
2023Optimistic Active Exploration of Dynamical Systemscited by 32
2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 28
2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26

38 more

2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 22
2023Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learningcited by 17
2022Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificatecited by 16
2023Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learningcited by 12
2024Constrained Reinforcement Learning with Smoothed Log Barrier Functioncited by 12
2022Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learningcited by 11
2022Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representationcited by 11
2024RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashescited by 10
2023State-Wise Safe Reinforcement Learning With Pixel Observationscited by 9
2023Decision-Making Under Uncertainty: Beyond Probabilitiescited by 8
2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
2023Gradient Shaping for Multi-Constraint Safe Reinforcement Learningcited by 7
2023Data-Efficient Task Generalization via Probabilistic Model-based Meta Reinforcement Learningcited by 6
2026Safe Exploration via Policy Priorscited by 6
2022Effects of Safety State Augmentation on Safe Explorationcited by 5
2024Leveraging Approximate Model-based Shielding for Probabilistic Safety Guarantees in Continuous Environmentscited by 5
2023Approximate Model-Based Shielding for Safe Reinforcement Learningcited by 4
2024Feasibility Consistent Representation Learning for Safe Reinforcement Learningcited by 3
2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
2024Flipping-based Policy for Chance-Constrained Markov Decision Processescited by 3
2025Object-Centric World Models for Causality-Aware Reinforcement Learningcited by 3
2023Approximate Shielding of Atari Agents for Safe Explorationcited by 2
2024Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimationcited by 2
2025PNAct: Crafting Backdoor Attacks in Safe Reinforcement Learningcited by 2
2023Bayesian inference for data-efficient, explainable, and safe robotic motion planning: A reviewcited by 1
2024Q-learning-based Model-free Safety Filtercited by 1
2025QuietPaw: Learning Quadrupedal Locomotion with Versatile Noise Preference Alignmentcited by 1
2025Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Reviewcited by 1
2025Safe and Near-Optimal Control with Online Dynamics Learningcited by 1
2026Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Modelscited by 1
2023Safe Deep RL for Intraoperative Planning of Pedicle Screw Placementcited by 0
2023Reinforcement Learning in a Safety-Embedded MDP with Trajectory Optimizationcited by 0
2024GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Modelcited by 0
2025Safe Explicable Policy Searchcited by 0
2025Safe Planning and Policy Optimization via World Model Learningcited by 0
2025Game-Theoretic Risk-Shaped Reinforcement Learning for Safe Autonomous Drivingcited by 0
2026Nightmare Dreamer: Dreaming About Unsafe States And Planning Aheadcited by 0
2026Safe Urban Traffic Control via Uncertainty-Aware Conformal Prediction and World-Model Reinforcement Learningcited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way