# Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement arXiv: https://arxiv.org/abs/2110.07067 ยท HTML: https://mlsanity.com/paper/2110.07067/ ## Key facts - Cited by 29 papers in the map (data through 14 Oct 2026). - Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. - Builds on 17 of its 25 references that are in the map. - First posted on arXiv 13 Oct 2021. ## Placed in - End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/) - Offline RL and imitation learning from data / [Imitation and offline RL for driving](https://mlsanity.com/offlinerl/imitation-and-offline-rl-for-driving/) ## Builds on (most cited first, 17 of 17) - [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013) - [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015) - [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013) - [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020) - [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018) - [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019) - [Input Convex Neural Networks](https://mlsanity.com/paper/1609.07152/) (2016) - [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017) - [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017) - [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017) - [Neural Lyapunov Control](https://mlsanity.com/paper/2005.00611/) (2020) - [Benchmarking Batch Deep Reinforcement Learning Algorithms](https://mlsanity.com/paper/1910.01708/) (2019) - [Learning Stable Deep Dynamics Models](https://mlsanity.com/paper/2001.06116/) (2020) - [Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning](https://mlsanity.com/paper/2001.08726/) (2020) - [Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving](https://mlsanity.com/paper/2104.07246/) (2021) - [Autonomous Driving using Safe Reinforcement Learning by Incorporating a Regret-based Human Lane-Changing Decision Model](https://mlsanity.com/paper/1910.04803/) (2019) - [Automated Driving Maneuvers under Interactive Environment based on Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09200/) (2018) ## Built on by (most cited first, 29 of 29) - [DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching](https://mlsanity.com/paper/2402.02439/) (2024) - [Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/2209.06620/) (2022) - [Rethinking Closed-loop Training for Autonomous Driving](https://mlsanity.com/paper/2306.15713/) (2023) - [An Empirical Study of Implicit Regularization in Deep Offline RL](https://mlsanity.com/paper/2207.02099/) (2022) - [A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem](https://mlsanity.com/paper/2305.17198/) (2023) - [AD4RL: Autonomous Driving Benchmarks for Offline Reinforcement Learning with Value-based Dataset](https://mlsanity.com/paper/2404.02429/) (2024) - [Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills](https://mlsanity.com/paper/2309.13614/) (2023) - [Transformers in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.05979/) (2023) - [Beyond Pessimism: Offline Learning in KL-regularized Games](https://mlsanity.com/paper/2604.06738/) (2026) - [Uncertainty-Aware Decision Transformer for Stochastic Driving Environments](https://mlsanity.com/paper/2309.16397/) (2023) - [Offline Reinforcement Learning for Road Traffic Control](https://mlsanity.com/paper/2201.02381/) (2022) - [Primal-Dual Spectral Representation for Off-policy Evaluation](https://mlsanity.com/paper/2410.17538/) (2024) - [Temporal Logic-Based Multi-Vehicle Backdoor Attacks against Offline RL Agents in End-to-end Autonomous Driving](https://mlsanity.com/paper/2509.16950/) (2025) - [Goal-Conditioned Reinforcement Learning from Sub-Optimal Data on Metric Spaces](https://mlsanity.com/paper/2402.10820/) (2024) - [Offline Reinforcement Learning with Domain-Unlabeled Data](https://mlsanity.com/paper/2404.07465/) (2024) - [Wavelet Fourier Diffuser: Frequency-Aware Diffusion Model for Reinforcement Learning](https://mlsanity.com/paper/2509.19305/) (2025) - [Learning to Control Autonomous Fleets from Observation via Offline Reinforcement Learning](https://mlsanity.com/paper/2302.14833/) (2023) - [Urban-Focused Multi-Task Offline Reinforcement Learning with Contrastive Data Sharing](https://mlsanity.com/paper/2406.14054/) (2024) - [Soft Conflict-Resolution Decision Transformer for Offline Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2511.13133/) (2025) - [Contrastive Diffuser: Planning Towards High Return States via Contrastive Learning](https://mlsanity.com/paper/2402.02772/) (2024) - [State-Constrained Offline Reinforcement Learning](https://mlsanity.com/paper/2405.14374/) (2024) - [Enhanced Safety in Autonomous Driving: Integrating Latent State Diffusion Model for End-to-End Navigation](https://mlsanity.com/paper/2407.06317/) (2024) - [Offline Reinforcement Learning using Human-Aligned Reward Labeling for Autonomous Emergency Braking in Occluded Pedestrian Crossing](https://mlsanity.com/paper/2504.08704/) (2025) - [Inference Time Policy Optimization for Offline RL with Differentiable World Models](https://mlsanity.com/paper/2603.22430/) (2026) - [Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RL](https://mlsanity.com/paper/2502.12391/) (2025) - [RAD: Retrieval High-quality Demonstrations to Enhance Decision-making](https://mlsanity.com/paper/2507.15356/) (2025) - [Safe-RULE: Safe Reinforcement UnLEarning](https://mlsanity.com/paper/2606.09559/) (2026) - [Learning from the Test: Self-Referential Differential Testing for Deep RL Agents](https://mlsanity.com/paper/2608.22284/) (2026) - [DaCe-DT: Data-Centric Offline Multi-Task Reinforcement Learning via Adaptive Prompts and Trajectory Correction for Heterogeneous Tasks](https://mlsanity.com/paper/2610.11085/) (2026) ## Abstract (arXiv) Reinforcement learning (RL) is a powerful data-driven control method that has been largely explored in autonomous driving tasks. However, conventional RL approaches learn control policies through trial-and-error interactions with the environment and therefore may cause disastrous consequences such as collisions when testing in real-world traffic. Offline RL has recently emerged as a promising framework to learn effective policies from previously-collected, static datasets without the requirement of active interactions, making it especially appealing for autonomous driving applications. Despite promising, existing offline RL algorithms such as Batch-Constrained deep Q-learning (BCQ) generally lead to rather conservative policies with limited exploration efficiency. To address such issues, this paper presents an enhanced BCQ algorithm by employing a learnable parameter noise scheme in the perturbation model to increase the diversity of observed actions. In addition, a Lyapunov-based safety enhancement strategy is incorporated to constrain the explorable state space within a safe region. Experimental results in highway and parking traffic scenarios show that our approach outperforms the conventional RL method, as well as state-of-the-art offline RL algorithms.