mlsanityData to 14 Oct 2026

FOSP: Fine-tuning Offline Safe Policy through World Models

arXiv 2407.04942 ↗First posted 6 Jul 2024cs.RO cs.LGcited by 3

On the map: World models → Latent dynamics models and model-based RL

also in 2 more
Abstract (arXiv)

Offline Safe Reinforcement Learning (RL) seeks to address safety constraints by learning from static datasets and restricting exploration. However, these approaches heavily rely on the dataset and struggle to generalize to unseen scenarios safely. In this paper, we aim to improve safety during the deployment of vision-based robotic tasks through online fine-tuning an offline pretrained policy. To facilitate effective fine-tuning, we introduce model-based RL, which is known for its data efficiency. Specifically, our method employs in-sample optimization to improve offline training efficiency while incorporating reachability guidance to ensure safety. After obtaining an offline safe policy, a safe policy expansion approach is leveraged for online fine-tuning. The performance of our method is validated on simulation benchmarks with five vision-only tasks and through real-world robot deployment using limited data. It demonstrates that our approach significantly improves the generalization of offline policies to unseen safety-constrained scenarios. To the best of our knowledge, this is the first work to explore offline-to-online RL for safe generalization tasks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 53 of its 60 references that are in the map. First posted on arXiv 6 Jul 2024.

Builds on

53 of 60 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Finetuning Offline World Models in the Real World24 Oct
  2. 2023SafeDreamer: Safe Reinforcement Learning with World Models14 Jul
  3. 2023Mastering Diverse Domains through World Models10 Jan
  4. 2022Constrained Policy Optimization via Bayesian World Models24 Jan
  5. 2020Mastering Atari with Discrete World Models5 Oct
  6. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec

Other references · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2017Constrained Policy Optimizationcited by 983
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
35 more
  1. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  4. 2018Reward Constrained Policy Optimizationcited by 417
  5. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  7. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  8. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  9. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  10. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  11. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  12. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 175
  13. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  14. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  15. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  16. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  17. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  18. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 95
  19. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  20. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  21. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 69
  22. 2020Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functionscited by 68
  23. 2022Reachability Constrained Reinforcement Learningcited by 59
  24. 2020Learning Off-Policy with Online Planningcited by 53
  25. 2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 37
  26. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  27. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  28. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 33
  29. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  30. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 28
  31. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RLcited by 24
  32. 2022On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrationscited by 19
  33. 2022Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificatecited by 18
  34. 2022Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representationcited by 11
  35. 2024Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025DMWM: Dual-Mind World Model with Long-Term Imaginationcited by 10
  2. 2025VLM-SAFE: Vision-Language Model-Guided Safety-Aware Reinforcement Learning with World Models for Autonomous Drivingcited by 9
  3. 2026SafeDojo: Safe Reinforcement Learning for VLA via Interactive World Modelcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
  2. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning
  3. 2025Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RL
  4. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
  5. 2025Safe Reinforcement Learning with Minimal Supervision
  6. 2025Skill-based Safe Reinforcement Learning with Risk Planning
  7. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  8. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning