mlsanityData to 7 Oct 2026

RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashes

arXiv 2405.04714 ↗First posted 7 May 2024cs.RO cs.AI cs.LGcited by 10

On the map: End-to-end autonomous driving → Reinforcement learning for driving

Abstract (arXiv)

Reinforcement learning provides an appealing framework for robotic control due to its ability to learn expressive policies purely through real-world interaction. However, this requires addressing real-world constraints and avoiding catastrophic failures during training, which might severely impede both learning progress and the performance of the final policy. In many robotics settings, this amounts to avoiding certain "unsafe" states. The high-speed off-road driving task represents a particularly challenging instantiation of this problem: a high-return policy should drive as aggressively and as quickly as possible, which often requires getting close to the edge of the set of "safe" states, and therefore places a particular burden on the method to avoid frequent failures. To both learn highly performant policies and avoid excessive failures, we propose a reinforcement learning framework that combines risk-sensitive control with an adaptive action space curriculum. Furthermore, we show that our risk-sensitive objective automatically avoids out-of-distribution states when equipped with an estimator for epistemic uncertainty. We implement our algorithm on a small-scale rally car and show that it is capable of learning high-speed policies for a real-world off-road driving task. We show that our method greatly reduces the number of safety violations during the training process, and actually leads to higher-performance policies in both driving and non-driving simulation environments with similar challenges.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. Builds on 34 of its 55 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

34 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing19 Apr

Other references · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2017Constrained Policy Optimizationcited by 928
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  10. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  11. 2018Reward Constrained Policy Optimizationcited by 397
  12. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
21 more
  1. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  2. 2021Ensemble deep learning: A reviewcited by 283
  3. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  4. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
  5. 2018Safe Exploration in Continuous Action Spacescited by 237
  6. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  7. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  8. 2020Uncertainty Quantification and Deep Ensemblescited by 138
  9. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 109
  10. 2020Conservative Safety Critics for Explorationcited by 107
  11. 2014Algorithms for CVaR Optimization in MDPscited by 98
  12. 2017Information Theoretic Model Predictive Control: Theory and Applications to Autonomous Drivingcited by 81
  13. 2020Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functionscited by 61
  14. 2022Constrained Policy Optimization via Bayesian World Modelscited by 55
  15. 2019Worst Cases Policy Gradientscited by 52
  16. 2020Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimizationcited by 34
  17. 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
  18. 2023Solving Stabilize-Avoid Optimal Control via Epigraph Form and Deep Reinforcement Learningcited by 24
  19. 2020High-speed Autonomous Drifting with Deep Reinforcement Learningcited by 20
  20. 2013Scaling Up Robust MDPs by Reinforcement Learningcited by 16
  21. 2021Risk-sensitive safety analysis using Conditional Value-at-Riskcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 4.

10 papers.

  1. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  2. 2026ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Controlcited by 5
  3. 2024Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learningcited by 2
  4. 2025A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspectivecited by 2
  5. 2025Neural Fidelity Calibration for Informative Sim-to-Real Adaptationcited by 1
  6. 2025Situationally-Aware Dynamics Learningcited by 1
  7. 2025Scalable Offline Metrics for Autonomous Drivingcited by 0
  8. 2026Solving Parameter-Robust Avoid Problems with Unknown Feasibility using Reinforcement Learningcited by 0
  9. 2026A Noise-Robust Elicit-to-Optimize Framework for Distortion Riskmetrics via Inverse Reinforcement Learningcited by 0
  10. 2026BCPPO: Bachelier-Inspired Constrained Proximal Policy Optimization for Tail-Risk-Aware Safe Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Minimizing Safety Interference for Safe and Comfortable Automated Driving with Distributional Reinforcement Learning
  2. 2023Pay Attention to How You Drive: Safe and Adaptive Model-Based Reinforcement Learning for Off-Road Driving
  3. 2022SafeRL-Kit: Evaluating Efficient Reinforcement Learning Methods for Safe Autonomous Driving
  4. 2026A Risk-Sensitive and Uncertainty-Aware Decision-Making and Control Framework for Safe and Robust Autonomous Driving
  5. 2025Balancing Progress and Safety: A Novel Risk-Aware Objective for RL in Autonomous Driving
  6. 2025AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Models
  7. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
  8. 2026Behavior-Constrained Reinforcement Learning with Receding-Horizon Credit Assignment for High-Performance Control ↗