# RoXDrive: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving via Action-Faithful Rollouts arXiv: https://arxiv.org/abs/2609.36851 ยท HTML: https://mlsanity.com/paper/2609.36851/ ## Key facts - No citing papers in the map yet (data through 14 Oct 2026). - Placed in World models for autonomous driving, one of 11 directions in World models. - Builds on 32 of its 70 references that are in the map. - First posted on arXiv 29 Sep 2026. ## Placed in - World models / [World models for autonomous driving](https://mlsanity.com/wm/world-models-for-autonomous-driving/) - End-to-end autonomous driving / [Imitation-learning end-to-end planners](https://mlsanity.com/driving/imitation-learning-end-to-end-planners/) - Offline RL and imitation learning from data / [Imitation and offline RL for driving](https://mlsanity.com/offlinerl/imitation-and-offline-rl-for-driving/) - Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/) ## Builds on (most cited first, 30 of 32) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022) - [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025) - [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025) - [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025) - [World Models](https://mlsanity.com/paper/1803.10122/) (2018) - [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016) - [ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving](https://mlsanity.com/paper/2506.08052/) (2025) - [Enhancing End-to-End Autonomous Driving with Latent World Model](https://mlsanity.com/paper/2406.08481/) (2024) - [FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving](https://mlsanity.com/paper/2505.17685/) (2025) - [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026) - [DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving](https://mlsanity.com/paper/2510.12796/) (2025) - [DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving](https://mlsanity.com/paper/2503.07656/) (2025) - [Epona: Autoregressive Diffusion World Model for Autonomous Driving](https://mlsanity.com/paper/2506.24113/) (2025) - [AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoning](https://mlsanity.com/paper/2503.07608/) (2025) - [RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning](https://mlsanity.com/paper/2502.13144/) (2025) - [Generalized Trajectory Scoring for End-to-end Multimodal Planning](https://mlsanity.com/paper/2506.06664/) (2025) - [DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving](https://mlsanity.com/paper/2512.07745/) (2025) - [Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)](https://mlsanity.com/paper/2505.16394/) (2025) - [From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction](https://mlsanity.com/paper/2510.19654/) (2025) - [Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving](https://mlsanity.com/paper/2603.24581/) (2026) - [LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving](https://mlsanity.com/paper/2603.01928/) (2026) - [DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning](https://mlsanity.com/paper/2604.01765/) (2026) - [Latent Chain-of-Thought World Modeling for End-to-End Driving](https://mlsanity.com/paper/2512.10226/) (2025) - [ResWorld: Temporal Residual World Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2602.10884/) (2026) - [Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving](https://mlsanity.com/paper/2603.27287/) (2026) - [MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving](https://mlsanity.com/paper/2602.21952/) (2026) - [X-World: Controllable Ego-Centric Multi-Camera World Models for Scalable End-to-End Driving](https://mlsanity.com/paper/2603.19979/) (2026) ## Abstract (arXiv) End-to-end autonomous driving policies are commonly trained via imitation learning on logged demonstrations without observing the consequences of their own actions, leading to causal confusion in closed-loop real-world deployment. To address this issue, reinforcement learning (RL) post-training offers a promising alternative by leveraging world models as interactive training environments to enable future scene generation for policy improvement. Nevertheless, existing approaches either rely on reconstruction-based simulators, offering limited counterfactual interaction, or adopt synthetic simulators to enable long-horizon closed-loop interaction at the cost of a substantial sim-to-real gap. Recently, video world models have exhibited the ability to generate realistic multi-step future rollouts but may not faithfully reflect action conditions, resulting in action-vision mismatch. In this paper, we introduce RoXDrive, a plug-and-play closed-loop RL framework that enables reliable policy optimization by identifying action-faithful world-model rollouts, consisting of two stages: 1) Model pre-training: In addition to imitation-based policy pre-training, we devise an Action-Vision Faithfulness Evaluator for inverse dynamics estimation with our geometry-aware auxiliary trajectory supervision, enabling long-horizon assessment of whether visual dynamics faithfully reflect the conditioning ego actions. 2) Action-faithful RL post-training: Agents iteratively interact with world models to form long-horizon scene rollouts, retaining only action-faithful ones for dense safety-aware scoring and scene-level closed-loop RL post-training. Extensive experiments on nuScenes and an in-house dataset with over 130K training scenarios demonstrate consistent gains across planners, reducing safety violations by 27.6% with DiffusionDrive on nuScenes and 33.7% with Qwen3-VL on the internal data.