mlsanityData to 14 Oct 2026

Beyond Safe Execution: Safe Online Reinforcement Learning for Autonomous Driving with Value Calibration and Adaptive Guidance

arXiv 2610.17799 ↗First posted 13 Oct 2026cs.ROcited by 0

On the map: End-to-end autonomous driving → Reinforcement learning for driving

also in 1 more
Abstract (arXiv)

Online reinforcement learning (RL) enables driving policies to adapt to changing deployment conditions, but requires safe exploration throughout training. Safety filtering alone does not ensure stable policy improvement or reduced dependence on corrections. We propose Safe Online Adaptive Refinement (SOAR-Flow), a framework for safe and stable online fine-tuning of offline-initialized Flow-based driving policies. A control barrier function (CBF)--model predictive control (MPC) filter corrects potentially unsafe actions before execution. To support stable policy updates, we introduce Gaussian-return-aware value calibration based on calibrated Q-learning (Cal-QL), using recorded behavior returns and predicted return variability to smooth conservative critic updates and mitigate policy degradation. Paired corrective learning teaches feasible action adjustments, while an intervention-to-go predictor encourages decisions requiring fewer future corrections. Its predictions also regulate guidance strength, allowing recovery supervision to weaken as predicted dependence decreases. Together with gradually reduced calibration regularization, this supports continued task improvement while preserving hard execution constraints. We analyze conditions for safe execution at control steps and diminishing auxiliary learning influences. We evaluate SOAR-Flow in simulation and on an uncrewed ground vehicle (UGV). The results demonstrate improved driving performance and reduced reliance on safety corrections during protected online fine-tuning. The demonstration video and project overview are available at https://lzqw.github.io/SOAR-Flow/.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. Builds on 22 of its 23 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

22 of 23 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025From Learning to Mastery: Achieving Safe and Efficient Real-World Autonomous Driving with Human-In-The-Loop Reinforcement Learning7 Oct
  2. 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization17 Feb
  3. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning26 Sep

Other references · 19

  1. 2022Flow Matching for Generative Modelingcited by 5,702
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  5. 2017Constrained Policy Optimizationcited by 983
  6. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  7. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  8. 2025Flow Q-Learningcited by 184
  9. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 175
  10. 2018A predictive safety filter for learning-based control of constrained nonlinear dynamical systemscited by 118
  11. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 85
  12. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 59
7 more
  1. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 29
  2. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 26
  3. 2023Distributional Soft Actor-Critic with Three Refinementscited by 25
  4. 2024Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agentscited by 17
  5. 2025CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functionscited by 17
  6. 2025SafeFlow: Safe Robot Motion Planning with Flow Matching via Control Barrier Functionscited by 13
  7. 2026Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulationcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
  2. 2022SafeRL-Kit: Evaluating Efficient Reinforcement Learning Methods for Safe Autonomous Driving
  3. 2025ADReFT: Adaptive Decision Repair for Safe Autonomous Driving via Reinforcement Fine-Tuning
  4. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  5. 2021Minimizing Safety Interference for Safe and Comfortable Automated Driving with Distributional Reinforcement Learning
  6. 2026Threat-guided Policy-aware Scene Perturbation for Safe Autonomous Driving with Online Reinforcement Learning
  7. 2025Safe Reinforcement Learning with a Predictive Safety Filter for Motion Planning and Control: A Drifting Vehicle Example
  8. 2026A Risk-Sensitive and Uncertainty-Aware Decision-Making and Control Framework for Safe and Robust Autonomous Driving