# AD4RL: Autonomous Driving Benchmarks for Offline Reinforcement Learning with Value-based Dataset

arXiv: https://arxiv.org/abs/2404.02429 · HTML: https://mlsanity.com/paper/2404.02429/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)
- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 25 of 25)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble](https://mlsanity.com/paper/2110.01548/) (2021)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Deep learning in agriculture: A survey](https://mlsanity.com/paper/1807.11809/) (2018)
- [PLAS: Latent Action Space for Offline Reinforcement Learning](https://mlsanity.com/paper/2011.07213/) (2020)
- [CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving](https://mlsanity.com/paper/1807.03776/) (2018)
- [Flow: A Modular Learning Framework for Mixed Autonomy Traffic](https://mlsanity.com/paper/1710.05465/) (2017)
- [Probabilistic 3D Multi-Modal, Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2012.13755/) (2020)
- [A Workflow for Offline Model-Free Robotic Reinforcement Learning](https://mlsanity.com/paper/2109.10813/) (2021)
- [When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning](https://mlsanity.com/paper/2206.13464/) (2022)
- [Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement](https://mlsanity.com/paper/2110.07067/) (2021)
- [Safe Real-World Autonomous Driving by Learning to Predict and Plan with a Mixture of Experts](https://mlsanity.com/paper/2211.02131/) (2022)
- [Simulation to Scaled City: Zero-Shot Policy Transfer for Traffic Control via Autonomous Vehicles](https://mlsanity.com/paper/1812.06120/) (2018)
- [Control-Aware Prediction Objectives for Autonomous Driving](https://mlsanity.com/paper/2204.13319/) (2022)

## Built on by (most cited first, 15 of 15)

- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces](https://mlsanity.com/paper/2410.15698/) (2024)
- [Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning](https://mlsanity.com/paper/2505.13144/) (2025)
- [Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2410.17373/) (2024)
- [Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformers](https://mlsanity.com/paper/2504.04395/) (2025)
- [Temporal Logic-Based Multi-Vehicle Backdoor Attacks against Offline RL Agents in End-to-end Autonomous Driving](https://mlsanity.com/paper/2509.16950/) (2025)
- [A Recipe for Stable Offline Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2603.08399/) (2026)
- [Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments](https://mlsanity.com/paper/2606.03698/) (2026)
- [Offline Reinforcement Learning using Human-Aligned Reward Labeling for Autonomous Emergency Braking in Occluded Pedestrian Crossing](https://mlsanity.com/paper/2504.08704/) (2025)
- [PyTupli: A Scalable Infrastructure for Collaborative Offline Reinforcement Learning Projects](https://mlsanity.com/paper/2505.16754/) (2025)
- [LLM-Driven Policy Diffusion: Enhancing Generalization in Offline Reinforcement Learning](https://mlsanity.com/paper/2509.00347/) (2025)
- [Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment](https://mlsanity.com/paper/2601.22823/) (2026)
- [From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving](https://mlsanity.com/paper/2603.17714/) (2026)
- [Simple Actors and Deep Critics for Scalable Reinforcement Learning](https://mlsanity.com/paper/2608.26659/) (2026)
- [A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platform](https://mlsanity.com/paper/2610.00926/) (2026)

## Abstract (arXiv)

Offline reinforcement learning has emerged as a promising technology by enhancing its practicality through the use of pre-collected large datasets. Despite its practical benefits, most algorithm development research in offline reinforcement learning still relies on game tasks with synthetic datasets. To address such limitations, this paper provides autonomous driving datasets and benchmarks for offline reinforcement learning research. We provide 19 datasets, including real-world human driver's datasets, and seven popular offline reinforcement learning algorithms in three realistic driving scenarios. We also provide a unified decision-making process model that can operate effectively across different scenarios, serving as a reference framework in algorithm design. Our research lays the groundwork for further collaborations in the community to explore practical aspects of existing reinforcement learning methods. Dataset and codes can be found in https://sites.google.com/view/ad4rl.