mlsanityData to 7 Oct 2026

Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environments

arXiv 2512.18662 ↗First posted 21 Dec 2025cs.RO cs.CVcited by 0

On the map: End-to-end autonomous driving → Imitation-learning end-to-end planners

also in 1 more
Abstract (arXiv)

End-to-end (E2E) autonomous driving models that take only camera images as input and directly predict a future trajectory are appealing for their computational efficiency and potential for improved generalization via unified optimization; however, persistent failure modes remain due to reliance on imitation learning (IL). While online reinforcement learning (RL) could mitigate IL-induced issues, the computational burden of neural rendering-based simulation and large E2E networks renders iterative reward and hyperparameter tuning costly. We introduce a camera-only E2E offline RL framework that performs no additional exploration and trains solely on a fixed simulator dataset. Offline RL offers strong data efficiency and rapid experimental iteration, yet is susceptible to instability from overestimation on out-of-distribution (OOD) actions. To address this, we construct pseudo ground-truth trajectories from expert driving logs and use them as a behavior regularization signal, suppressing imitation of unsafe or suboptimal behavior while stabilizing value learning. Training and closed-loop evaluation are conducted in a neural rendering environment learned from the public nuScenes dataset. Empirically, the proposed method achieves substantial improvements in collision rate and route completion compared with IL baselines. Our code is available at https://github.com/ToyotaInfoTech/PEBC.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. Builds on 53 of its 60 references that are in the map. First posted on arXiv 21 Dec 2025.

Builds on

53 of 60 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning18 Feb
  2. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving22 Nov
  3. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning20 Feb
  4. 2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios21 Dec
  5. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coach18 Aug

Other references · 48

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  6. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  12. 2022Planning-oriented Autonomous Drivingcited by 1,068
36 more
  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  5. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  6. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  7. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  9. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  10. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  12. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  13. 2019Causal Confusion in Imitation Learningcited by 286
  14. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  15. 2024Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillationcited by 244
  16. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  17. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  18. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  19. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 181
  20. 2025Flow Q-Learningcited by 162
  21. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  22. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  23. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  24. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 146
  25. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 125
  26. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  27. 2021Learning to drive from a world on railscited by 114
  28. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  29. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  30. 2020Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learningcited by 99
  31. 2018CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-drivingcited by 96
  32. 2025Robust Autonomy Emerges from Self-Playcited by 64
  33. 2024NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Drivingcited by 61
  34. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 49
  35. 2024Solving Motion Planning Tasks with a Scalable Generative Modelcited by 40
  36. 2024LORD: Large Models based Opposite Reward Design for Autonomous Drivingcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024From Imitation to Exploration: End-to-end Autonomous Driving based on World Model
  2. 2026RoXDrive: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving via Action-Faithful Rollouts
  3. 2026TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations
  4. 2026PerlAD: Towards Enhanced Closed-loop End-to-end Autonomous Driving with Pseudo-simulation-based Reinforcement Learning
  5. 2025From Imitation to Optimization: A Comparative Study of Offline Learning for Autonomous Driving
  6. 2026DriveZero: End-to-End Driving Beyond Human Demonstrations
  7. 2026Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving
  8. 2023Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills