# Echoes of Socratic Doubt: Embracing Uncertainty in Calibrated Evidential Reinforcement Learning

arXiv: https://arxiv.org/abs/2402.07107 · HTML: https://mlsanity.com/paper/2402.07107/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 11 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 17 of 17)

- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification](https://mlsanity.com/paper/2011.09588/) (2020)
- [Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation](https://mlsanity.com/paper/2110.03051/) (2021)
- [Estimating Risk and Uncertainty in Deep Reinforcement Learning](https://mlsanity.com/paper/1905.09638/) (2019)
- [A Review of Uncertainty for Deep Reinforcement Learning](https://mlsanity.com/paper/2208.09052/) (2022)
- [Efficient exploration with Double Uncertain Value Networks](https://mlsanity.com/paper/1711.10789/) (2017)
- [Lightweight, Uncertainty-Aware Conformalized Visual Odometry](https://mlsanity.com/paper/2303.02207/) (2023)
- [Deep Evidential Learning for Bayesian Quantile Regression](https://mlsanity.com/paper/2308.10650/) (2023)

## Built on by (most cited first, 3 of 3)

- [Learnable Conformal Prediction with Context-Aware Nonconformity Functions for Robotic Planning and Perception](https://mlsanity.com/paper/2509.21955/) (2025)
- [Auditing the Risk Claims of Distributional Reinforcement Learning](https://mlsanity.com/paper/2607.11607/) (2026)
- [Uncertainty-Driven Replay Memory for Reinforcement Learning](https://mlsanity.com/paper/2608.29860/) (2026)

## Abstract (arXiv)

We present a novel statistical approach to incorporating uncertainty awareness in model-free distributional reinforcement learning involving quantile regression-based deep Q networks. The proposed algorithm, textitCalibrated Evidential Quantile Regression in Deep Q Networks (CEQR-DQN), aims to address key challenges associated with separately estimating aleatoric and epistemic uncertainty in stochastic environments. It combines deep evidential learning with quantile calibration based on principles of conformal inference to provide explicit, sample-free computations of textitglobal uncertainty as opposed to textitlocal estimates based on simple variance, overcoming limitations of traditional methods in computational and statistical efficiency and handling of out-of-distribution (OOD) observations. Tested on a suite of miniaturized Atari games (i.e., MinAtar), CEQR-DQN is shown to surpass similar existing frameworks in scores and learning speed. Its ability to rigorously evaluate uncertainty improves exploration strategies and can serve as a blueprint for other algorithms requiring uncertainty awareness.