# Uncertainty-Aware and Temporally Regulated Expert Advice in Reinforcement Learning for Autonomous Driving

arXiv: https://arxiv.org/abs/2605.30576 · HTML: https://mlsanity.com/paper/2605.30576/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 28 May 2026.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)
- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 29 of 29)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [HG-DAgger: Interactive Imitation Learning with Human Experts](https://mlsanity.com/paper/1810.02890/) (2018)
- [GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Driving](https://mlsanity.com/paper/2111.08575/) (2021)
- [A Review of Uncertainty for Deep Reinforcement Learning](https://mlsanity.com/paper/2208.09052/) (2022)
- [Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Map-less Navigation by Leveraging Prior Demonstrations](https://mlsanity.com/paper/1805.07095/) (2018)
- [CaRL: Learning Scalable Planning Policies with Simple Rewards](https://mlsanity.com/paper/2504.17838/) (2025)
- [HAIM-DRL: Enhanced Human-in-the-loop Reinforcement Learning for Safe and Efficient Autonomous Driving](https://mlsanity.com/paper/2401.03160/) (2024)
- [Safe Driving via Expert Guided Policy Optimization](https://mlsanity.com/paper/2110.06831/) (2021)
- [AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers](https://mlsanity.com/paper/1909.04121/) (2019)
- [Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms](https://mlsanity.com/paper/2310.03225/) (2023)
- [Learning from Active Human Involvement through Proxy Value Propagation](https://mlsanity.com/paper/2502.03369/) (2025)
- [Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving](https://mlsanity.com/paper/2105.10266/) (2021)
- [Addressing Inherent Uncertainty: Risk-Sensitive Behavior Generation for Automated Driving using Distributional Reinforcement Learning](https://mlsanity.com/paper/2102.03119/) (2021)
- [Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions](https://mlsanity.com/paper/2305.09793/) (2023)
- [Minimizing Safety Interference for Safe and Comfortable Automated Driving with Distributional Reinforcement Learning](https://mlsanity.com/paper/2107.07316/) (2021)
- [Guarded Policy Optimization with Imperfect Online Demonstrations](https://mlsanity.com/paper/2303.01728/) (2023)
- [Student-Initiated Action Advising via Advice Novelty](https://mlsanity.com/paper/2010.00381/) (2020)
- [Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutions](https://mlsanity.com/paper/2409.13144/) (2024)
- [Value Functions are Control Barrier Functions: Verification of Safe Policies using Control Theory](https://mlsanity.com/paper/2306.04026/) (2023)
- [Safe Reinforcement Learning in Black-Box Environments via Adaptive Shielding](https://mlsanity.com/paper/2405.18180/) (2024)
- [IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuning](https://mlsanity.com/paper/2505.10442/) (2025)
- [Reducing Safety Interventions in Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2303.03339/) (2023)
- [Learning to Recover for Safe Reinforcement Learning](https://mlsanity.com/paper/2309.11907/) (2023)
- [Agent-Aware Training for Agent-Agnostic Action Advising in Deep Reinforcement Learning](https://mlsanity.com/paper/2311.16807/) (2023)

## Abstract (arXiv)

Exploration in reinforcement learning for autonomous driving is inherently unsafe: agents must experience novel behaviors to learn, yet exploration can lead to collisions or off-road driving. We propose an uncertainty-aware framework that leverages expert advice to guide exploration while avoiding long-term dependence. Advice is triggered when epistemic or aleatoric uncertainty exceeds adaptive thresholds derived from rolling buffers, ensuring advice evolves with the agent's confidence. A commitment-cooldown strategy with a stochastic early-stop heuristic regulates the duration and frequency of guidance, exposing the agent to coherent maneuvers without exhausting the advice budget. Expert and agent experiences are combined in a shared replay buffer within an off-policy implicit quantile network (IQN) backbone, enabling efficient reuse of expert trajectories. Experiments in CARLA show that our method outperforms the IQN baseline, improving success by 5-7% and reducing failures, demonstrating that risk-sensitive uncertainty coupled with regulated expert integration enables safer and more efficient exploration for sensor-based RL policy learning in unsignalized intersection navigation.