# SafeDreamer: Safe Reinforcement Learning with World Models

arXiv: https://arxiv.org/abs/2307.07176 · HTML: https://mlsanity.com/paper/2307.07176/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 33 of its 61 references that are in the map.
- First posted on arXiv 14 Jul 2023.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Responsive Safety in Reinforcement Learning by PID Lagrangian Methods](https://mlsanity.com/paper/2007.03964/) (2020)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model](https://mlsanity.com/paper/1907.00953/) (2019)
- [Projection-Based Constrained Policy Optimization](https://mlsanity.com/paper/2010.03152/) (2020)
- [Learning-based Model Predictive Control for Safe Exploration](https://mlsanity.com/paper/1803.08287/) (2018)
- [Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning](https://mlsanity.com/paper/2206.08686/) (2022)
- [A predictive safety filter for learning-based control of constrained nonlinear dynamical systems](https://mlsanity.com/paper/1812.05506/) (2018)
- [OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research](https://mlsanity.com/paper/2305.09304/) (2023)
- [Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentation](https://mlsanity.com/paper/2202.06558/) (2022)
- [Safe Reinforcement Learning by Imagining the Near Future](https://mlsanity.com/paper/2202.07789/) (2022)
- [Constrained Update Projection Approach to Safe Policy Optimization](https://mlsanity.com/paper/2209.07089/) (2022)
- [Constrained Policy Optimization via Bayesian World Models](https://mlsanity.com/paper/2201.09802/) (2022)
- [Learning Off-Policy with Online Planning](https://mlsanity.com/paper/2008.10066/) (2020)
- [Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm](https://mlsanity.com/paper/2210.07573/) (2022)
- [Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learning](https://mlsanity.com/paper/2111.07695/) (2021)
- [AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learning](https://mlsanity.com/paper/2301.10339/) (2023)

## Built on by (most cited first, 30 of 54)

- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond](https://mlsanity.com/paper/2405.03520/) (2024)
- [World Models for Autonomous Driving: An Initial Survey](https://mlsanity.com/paper/2403.02622/) (2024)
- [The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control](https://mlsanity.com/paper/2412.03568/) (2024)
- [LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoning](https://mlsanity.com/paper/2403.11552/) (2024)
- [A Survey of World Models for Autonomous Driving](https://mlsanity.com/paper/2501.11260/) (2025)
- [A Survey: Learning Embodied Intelligence from Physical Simulators and World Models](https://mlsanity.com/paper/2507.00917/) (2025)
- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning](https://mlsanity.com/paper/2410.09486/) (2024)
- [Constrained Reinforcement Learning with Smoothed Log Barrier Function](https://mlsanity.com/paper/2403.14508/) (2024)
- [Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2405.13863/) (2024)
- [Latent Chain-of-Thought World Modeling for End-to-End Driving](https://mlsanity.com/paper/2512.10226/) (2025)
- [Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulation](https://mlsanity.com/paper/2601.07821/) (2026)
- [Is the Future Compatible? Diagnosing Dynamic Consistency in World Action Models](https://mlsanity.com/paper/2605.07514/) (2026)
- [OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2407.14653/) (2024)
- [Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only](https://mlsanity.com/paper/2410.11055/) (2024)
- [Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Control](https://mlsanity.com/paper/2502.03640/) (2025)
- [SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer](https://mlsanity.com/paper/2509.18648/) (2025)
- [Robust Training of Federated Models with Extremely Label Deficiency](https://mlsanity.com/paper/2402.14430/) (2024)
- [PIGDreamer: Privileged Information Guided World Models for Safe Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2508.02159/) (2025)
- [WMAttack: Automated Attack Search for Adversarial Evaluation of World-Model Agents](https://mlsanity.com/paper/2605.23220/) (2026)
- [Solving Multi-Agent Safe Optimal Control with Distributed Epigraph Form MARL](https://mlsanity.com/paper/2504.15425/) (2025)
- [Safe Exploration via Policy Priors](https://mlsanity.com/paper/2601.19612/) (2026)
- [A Dynamic Safety Shield for Safe and Efficient Reinforcement Learning of Navigation Tasks](https://mlsanity.com/paper/2412.04153/) (2024)
- [Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems](https://mlsanity.com/paper/2507.14850/) (2025)
- [WARPD: World model Assisted Reactive Policy Diffusion](https://mlsanity.com/paper/2410.14040/) (2024)
- [Safety Representations for Safer Policy Learning](https://mlsanity.com/paper/2502.20341/) (2025)
- [Object-Centric World Models for Causality-Aware Reinforcement Learning](https://mlsanity.com/paper/2511.14262/) (2025)
- [World Model Robustness via Surprise Recognition](https://mlsanity.com/paper/2512.01119/) (2025)
- [Morphology-Conditioned World Model for Cross-Embodiment Quadrupedal Locomotion](https://mlsanity.com/paper/2604.08780/) (2026)

## Abstract (arXiv)

The deployment of Reinforcement Learning (RL) in real-world applications is constrained by its failure to satisfy safety criteria. Existing Safe Reinforcement Learning (SafeRL) methods, which rely on cost functions to enforce safety, often fail to achieve zero-cost performance in complex scenarios, especially vision-only tasks. These limitations are primarily due to model inaccuracies and inadequate sample efficiency. The integration of the world model has proven effective in mitigating these shortcomings. In this work, we introduce SafeDreamer, a novel algorithm incorporating Lagrangian-based methods into world model planning processes within the superior Dreamer framework. Our method achieves nearly zero-cost performance on various tasks, spanning low-dimensional and vision-only input, within the Safety-Gymnasium benchmark, showcasing its efficacy in balancing performance and safety in RL tasks. Further details can be found in the code repository: urlhttps://github.com/PKU-Alignment/SafeDreamer.