# Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving

arXiv: https://arxiv.org/abs/2105.10266 · HTML: https://mlsanity.com/paper/2105.10266/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 21 May 2021.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding](https://mlsanity.com/paper/1511.02680/) (2015)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [Algorithms for CVaR Optimization in MDPs](https://mlsanity.com/paper/1406.3339/) (2014)
- [Virtual to Real Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1704.03952/) (2017)
- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics](https://mlsanity.com/paper/1710.11040/) (2017)
- [Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1811.06817/) (2018)
- [Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving](https://mlsanity.com/paper/1905.02680/) (2019)
- [Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning](https://mlsanity.com/paper/1803.10056/) (2018)
- [Navigating Occluded Intersections with Autonomous Vehicles using Deep Reinforcement Learning](https://mlsanity.com/paper/1705.01196/) (2017)
- [Reinforcement Learning with Uncertainty Estimation for Tactical Decision-Making in Intersections](https://mlsanity.com/paper/2006.09786/) (2020)
- [Tactical Decision-Making in Autonomous Driving by Reinforcement Learning with Uncertainty Estimation](https://mlsanity.com/paper/2004.10439/) (2020)
- [Learning Negotiating Behavior Between Cars in Intersections using Deep Q-Learning](https://mlsanity.com/paper/1810.10469/) (2018)
- [Addressing Inherent Uncertainty: Risk-Sensitive Behavior Generation for Automated Driving using Distributional Reinforcement Learning](https://mlsanity.com/paper/2102.03119/) (2021)

## Built on by (most cited first, 17 of 17)

- [A Unified Evaluation Framework for Epistemic Predictions](https://mlsanity.com/paper/2501.16912/) (2025)
- [Decision Making for Human-in-the-loop Robotic Agents via Uncertainty-Aware Reinforcement Learning](https://mlsanity.com/paper/2303.06710/) (2023)
- [Diverse Projection Ensembles for Distributional Reinforcement Learning](https://mlsanity.com/paper/2306.07124/) (2023)
- [Safe and Robust Reinforcement Learning: Principles and Practice](https://mlsanity.com/paper/2403.18539/) (2024)
- [Learning Ensembles of Vision-based Safety Control Filters](https://mlsanity.com/paper/2412.02029/) (2024)
- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [uSF: Learning Neural Semantic Field with Uncertainty](https://mlsanity.com/paper/2312.08012/) (2023)
- [Uncertainty-Aware Safety-Critical Decision and Control for Autonomous Vehicles at Unsignalized Intersections](https://mlsanity.com/paper/2505.19939/) (2025)
- [Universal Value-Function Uncertainties](https://mlsanity.com/paper/2505.21119/) (2025)
- [Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learning](https://mlsanity.com/paper/2408.02295/) (2024)
- [Uncertainty separation via ensemble quantile regression](https://mlsanity.com/paper/2412.13738/) (2024)
- [Game-Theoretic Risk-Shaped Reinforcement Learning for Safe Autonomous Driving](https://mlsanity.com/paper/2510.10960/) (2025)
- [Identifying Influential N-grams in Confidence Calibration via Regression Analysis](https://mlsanity.com/paper/2604.05757/) (2026)
- [Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration](https://mlsanity.com/paper/2605.03820/) (2026)
- [Uncertainty-Aware and Temporally Regulated Expert Advice in Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2605.30576/) (2026)
- [Auditing the Risk Claims of Distributional Reinforcement Learning](https://mlsanity.com/paper/2607.11607/) (2026)
- [A Risk-Sensitive and Uncertainty-Aware Decision-Making and Control Framework for Safe and Robust Autonomous Driving](https://mlsanity.com/paper/2609.09650/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) can be used to create a decision-making agent for autonomous driving. However, previous approaches provide only black-box solutions, which do not offer information on how confident the agent is about its decisions. An estimate of both the aleatoric and epistemic uncertainty of the agent's decisions is fundamental for real-world applications of autonomous driving. Therefore, this paper introduces the Ensemble Quantile Networks (EQN) method, which combines distributional RL with an ensemble approach, to obtain a complete uncertainty estimate. The distribution over returns is estimated by learning its quantile function implicitly, which gives the aleatoric uncertainty, whereas an ensemble of agents is trained on bootstrapped data to provide a Bayesian estimation of the epistemic uncertainty. A criterion for classifying which decisions that have an unacceptable uncertainty is also introduced. The results show that the EQN method can balance risk and time efficiency in different occluded intersection scenarios, by considering the estimated aleatoric uncertainty. Furthermore, it is shown that the trained agent can use the epistemic uncertainty information to identify situations that the agent has not been trained for and thereby avoid making unfounded, potentially dangerous, decisions outside of the training distribution.