# AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Models arXiv: https://arxiv.org/abs/2511.20325 ยท HTML: https://mlsanity.com/paper/2511.20325/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Placed in World models for autonomous driving, one of 11 directions in World models. - Builds on 42 of its 52 references that are in the map. - First posted on arXiv 25 Nov 2025. ## Placed in - World models / [World models for autonomous driving](https://mlsanity.com/wm/world-models-for-autonomous-driving/) - End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/) ## Builds on (most cited first, 30 of 42) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019) - [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019) - [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017) - [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022) - [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023) - [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022) - [NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking](https://mlsanity.com/paper/2406.15349/) (2024) - [Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability](https://mlsanity.com/paper/2405.17398/) (2024) - [DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2411.15139/) (2024) - [Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving](https://mlsanity.com/paper/2304.14365/) (2023) - [VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning](https://mlsanity.com/paper/2402.13243/) (2024) - [MagicDrive: Street View Generation with Diverse 3D Geometry Control](https://mlsanity.com/paper/2310.02601/) (2023) - [Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation](https://mlsanity.com/paper/2406.06978/) (2024) - [OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving](https://mlsanity.com/paper/2311.16038/) (2023) - [ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving](https://mlsanity.com/paper/2506.08052/) (2025) - [DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation](https://mlsanity.com/paper/2403.06845/) (2024) - [VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning](https://mlsanity.com/paper/2505.18719/) (2025) - [Enhancing End-to-End Autonomous Driving with Latent World Model](https://mlsanity.com/paper/2406.08481/) (2024) - [End-to-End Driving with Online Trajectory Evaluation via BEV World Model](https://mlsanity.com/paper/2504.01941/) (2025) - [GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories Generation in End-to-End Autonomous Driving](https://mlsanity.com/paper/2503.05689/) (2025) - [Epona: Autoregressive Diffusion World Model for Autonomous Driving](https://mlsanity.com/paper/2506.24113/) (2025) - [ADriver-I: A General World Model for Autonomous Driving](https://mlsanity.com/paper/2311.13549/) (2023) - [DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving](https://mlsanity.com/paper/2408.00415/) (2024) - [DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers](https://mlsanity.com/paper/2412.18607/) (2024) - [World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Model](https://mlsanity.com/paper/2507.00603/) (2025) - [RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning](https://mlsanity.com/paper/2502.13144/) (2025) - [OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving](https://mlsanity.com/paper/2405.20337/) (2024) - [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025) ## Built on by (most cited first, 14 of 14) - [DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Driving](https://mlsanity.com/paper/2602.06521/) (2026) - [Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future](https://mlsanity.com/paper/2512.16760/) (2025) - [UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving](https://mlsanity.com/paper/2604.02190/) (2026) - [DreamerAD: Efficient Reinforcement Learning via Latent World Model for Autonomous Driving](https://mlsanity.com/paper/2603.24587/) (2026) - [ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving](https://mlsanity.com/paper/2604.02714/) (2026) - [RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework](https://mlsanity.com/paper/2604.15308/) (2026) - [World Engine: Towards the Era of Post-Training for Autonomous Driving](https://mlsanity.com/paper/2606.19836/) (2026) - [CausalDrive: Real-time Causal World Models for Autonomous Driving](https://mlsanity.com/paper/2606.15341/) (2026) - [Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges](https://mlsanity.com/paper/2603.09086/) (2026) - [Distill to Think, Foresee to Act: Cognitive-Physical Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2605.21139/) (2026) - [DynFlowDrive: Flow-Based Dynamic World Modeling for Autonomous Driving](https://mlsanity.com/paper/2603.19675/) (2026) - [CRAFT: Counterfactual-to-Interactive Reinforcement Fine-Tuning for Driving Policies](https://mlsanity.com/paper/2605.04470/) (2026) - [Quo Vadis, World Modeling?](https://mlsanity.com/paper/2608.02713/) (2026) - [VITAL: From Visual Motion to Precise Action](https://mlsanity.com/paper/2610.20604/) (2026) ## Abstract (arXiv) End-to-end models for autonomous driving hold the promise of learning complex behaviors directly from sensor data, but face critical challenges in safety and handling long-tail events. Reinforcement Learning (RL) offers a promising path to overcome these limitations, yet its success in autonomous driving has been elusive. We identify a fundamental flaw hindering this progress: a deep seated optimistic bias in the world models used for RL. To address this, we introduce a framework for post-training policy refinement built around an Impartial World Model. Our primary contribution is to teach this model to be honest about danger. We achieve this with a novel data synthesis pipeline, Counterfactual Synthesis, which systematically generates a rich curriculum of plausible collisions and off-road events. This transforms the model from a passive scene completer into a veridical forecaster that remains faithful to the causal link between actions and outcomes. We then integrate this Impartial World Model into our closed-loop RL framework, where it serves as an internal critic. During refinement, the agent queries the critic to ``dream" of the outcomes for candidate actions. We demonstrate through extensive experiments, including on a new Risk Foreseeing Benchmark, that our model significantly outperforms baselines in predicting failures. Consequently, when used as a critic, it enables a substantial reduction in safety violations in challenging simulations, proving that teaching a model to dream of danger is a critical step towards building truly safe and intelligent autonomous agents.