mlsanityData to 7 Oct 2026

Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration

arXiv 2309.09408 ↗First posted 18 Sep 2023cs.RO cs.LGcited by 14

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Safe Reinforcement Learning (RL) aims to find a policy that achieves high rewards while satisfying cost constraints. When learning from scratch, safe RL agents tend to be overly conservative, which impedes exploration and restrains the overall performance. In many realistic tasks, e.g. autonomous driving, large-scale expert demonstration data are available. We argue that extracting expert policy from offline data to guide online exploration is a promising solution to mitigate the conserveness issue. Large-capacity models, e.g. decision transformers (DT), have been proven to be competent in offline policy learning. However, data collected in real-world scenarios rarely contain dangerous cases (e.g., collisions), which makes it prohibitive for the policies to learn safety concepts. Besides, these bulk policy networks cannot meet the computation speed requirements at inference time on real-world tasks such as autonomous driving. To this end, we propose Guided Online Distillation (GOLD), an offline-to-online safe RL framework. GOLD distills an offline DT policy into a lightweight policy network through guided online safe RL training, which outperforms both the offline DT policy and online safe RL algorithms. Experiments in both benchmark safe RL tasks and real-world driving tasks based on the Waymo Open Motion Dataset (WOMD) demonstrate that GOLD can successfully distill lightweight policies and solve decision-making problems in challenging safety-critical scenarios.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 45 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

36 of 45 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  9. 2017Constrained Policy Optimizationcited by 928
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  12. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
24 more
  1. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  2. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  3. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  4. 2015Policy Distillationcited by 414
  5. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 281
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  7. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  8. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  9. 2022Jump-Start Reinforcement Learningcited by 102
  10. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  11. 2019Distilling Policy Distillationcited by 81
  12. 2020Safe Reinforcement Learning via Curriculum Inductioncited by 73
  13. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  14. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  15. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  16. 2019DisCoRL: Continual Reinforcement Learning via Policy Distillationcited by 35
  17. 2021Spatio-Temporal Graph Dual-Attention Network for Multi-Agent Prediction and Trackingcited by 35
  18. 2023WOMD-LiDAR: Raw Sensor Dataset Benchmark for Motion Forecastingcited by 28
  19. 2019CMTS: Conditional Multiple Trajectory Synthesizer for Generating Safety-critical Driving Scenarioscited by 19
  20. 2021A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learningcited by 18
  21. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learningcited by 18
  22. 2022Offline-Online Learning of Deformation Model for Cable Manipulation with Graph Neural Networkscited by 17
  23. 2021Interaction-Aware Behavior Planning for Autonomous Vehicles Validated with Real Traffic Datacited by 4
  24. 2022Prim-LAfD: A Framework to Learn and Adapt Primitive-Based Skills from Demonstrations for Insertion Taskscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 4; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
  2. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  3. 2023Gradient Shaping for Multi-Constraint Safe Reinforcement Learningcited by 7
  4. 2024Adaptive Prediction Ensemble: Improving Out-of-Distribution Generalization of Motion Forecastingcited by 7
  5. 2024BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human Racing Gameplaycited by 3
  6. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
  7. 2024Physics-informed Imitative Reinforcement Learning for Real-world Drivingcited by 2
  8. 2025A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspectivecited by 2
  9. 2025Boundary-to-Region Supervision for Offline Safe Reinforcement Learningcited by 1
  10. 2025Online Optimization for Offline Safe Reinforcement Learningcited by 1
  11. 2026OnPoint: Offline-to-Online Multi-Level Distillation for Point-Supervised Online Temporal Action Localizationcited by 1
  12. 2025Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulationcited by 0
2 more
  1. 2026Imitation Learning from Human Motion Alone Does Not Guarantee Biomechanically Plausible Gait Kineticscited by 0
  2. 2026Safe-RULE: Safe Reinforcement UnLEarningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  2. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning
  3. 2024FOSP: Fine-tuning Offline Safe Policy through World Models
  4. 2021Safe Driving via Expert Guided Policy Optimization
  5. 2026Individual Control Barrier Functions-Guided Diffusion Model for Safe Offline Multi-Agent Reinforcement Learning
  6. 2024Offline Safe Reinforcement Learning Using Trajectory Classification
  7. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
  8. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model