mlsanityData to 14 Oct 2026

Towards Fast Safe Online Reinforcement Learning via Policy Finetuning

arXiv 2412.04426 ↗First posted 5 Dec 2024cs.LG cs.AIcited by 3

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

The high costs and risks involved in extensive environment interactions hinder the practical application of current online safe reinforcement learning (RL) methods. While offline safe RL addresses this by learning policies from static datasets, the performance therein is usually limited due to reliance on data quality and challenges with out-of-distribution (OOD) actions. Inspired by recent successes in offline-to-online (O2O) RL, it is crucial to explore whether offline safe RL can be leveraged to facilitate faster and safer online policy learning, a direction that has yet to be fully investigated. To fill this gap, we first demonstrate that naively applying existing O2O algorithms from standard RL would not work well in the safe RL setting due to two unique challenges: erroneous Q-estimations, resulted from offline-online objective mismatch and offline cost sparsity, and Lagrangian mismatch, resulted from difficulties in aligning Lagrange multipliers between offline and online policies. To address these challenges, we introduce Marvel, a novel framework for O2O safe RL, comprising two key components that work in concert: Value Pre-Alignment to align the Q-functions with the underlying truth before online learning, and Adaptive PID Control to effectively adjust the Lagrange multipliers during online finetuning. Extensive experiments demonstrate that Marvel significantly outperforms existing baselines in both reward maximization and safety constraint satisfaction. By introducing the first policy-finetuning based framework for O2O safe RL, which is compatible with many offline and online safe RL methods, our work has the great potential to advance the field towards more efficient and practical safe RL solutions.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 50 of its 64 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

50 of 64 references.

References · 50

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2017Constrained Policy Optimizationcited by 983
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  9. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  11. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 453
  12. 2018Reward Constrained Policy Optimizationcited by 417
38 more
  1. 2019Control Barrier Functions: Theory and Applicationscited by 372
  2. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  3. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  4. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  5. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  6. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  7. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  8. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  9. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  10. 2020Projection-Based Constrained Policy Optimizationcited by 203
  11. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 201
  12. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  13. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  14. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  15. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  16. 2022Jump-Start Reinforcement Learningcited by 112
  17. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  18. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  19. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  20. 2020Secure and Robust Machine Learning for Healthcare: A Surveycited by 80
  21. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  22. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 69
  23. 2020Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functionscited by 68
  24. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
  25. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
  26. 2022Reachability Constrained Reinforcement Learningcited by 59
  27. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 59
  28. 2022Offline RL Policies Should be Trained to be Adaptivecited by 52
  29. 2021safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Roboticscited by 50
  30. 2021AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scalecited by 44
  31. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  32. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  33. 2022CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learningcited by 20
  34. 2022On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrationscited by 19
  35. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learningcited by 15
  36. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstrationcited by 14
  37. 2020Learning a Low-dimensional Representation of a Safe Region for Safe Reinforcement Learning on Dynamical Systemscited by 6
  38. 2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026HIPO: Instruction Hierarchy via Constrained Reinforcement Learningcited by 3
  2. 2026Adaptive Control in Autonomous Driving via Real-Time Recurrent RLcited by 0
  3. 2026Information-Directed Offline-to-Online Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Online Optimization for Offline Safe Reinforcement Learning
  2. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  3. 2026Active Offline-to-Online Reinforcement Learning
  4. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  5. 2025Safe Reinforcement Learning with Minimal Supervision
  6. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
  7. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning
  8. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption