mlsanityData to 14 Oct 2026

Auditing the Risk Claims of Distributional Reinforcement Learning

arXiv 2607.11607 ↗First posted 13 Jul 2026cs.AI cs.LG stat.MLcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Distributional reinforcement learning agents learn full return distributions that are increasingly read at face value: for interpretability, risk-sensitive control, and safety monitoring. We ask a question theory anticipates but that has not been measured directly: are the risk claims of a trained distributional agent true? Our audit combines a decision-relevant screening metric (the excess Wasserstein gap between the top two actions, which equals the mass by which first-order stochastic dominance is violated), ground truth from snapshot-restart Monte Carlo, and a statistical harness (permutation nulls, bootstrap refutation, FDR control) without which the audit itself manufactures false conclusions. Across QR-DQN, C51, and IQN on MinAtar (33 runs), 40-95% of the strongest claimed risk trade-offs are refuted at 95% confidence, the placement of the strongest claims is statistically indistinguishable from truth-blind, and essentially no claim is confirmable: for these agents, the learned "risk" reflects a training artifact rather than environment stochasticity. The artifact is structural (fully formed early in training, uncorrelated with final score, idiosyncratic to each seed) and appears unchanged at full-Atari scale, with every top Breakout claim of a pretrained near-state-of-the-art QR-DQN refuted. Positive controls of known magnitude confirm 96-100% of real claims (correlation 0.89-0.92): the reading measures the agents, not the audit. Acting on the heads' CVaR advice at their most-flagged states ranges from beneficial to significantly worse than chance. Neither training for risk nor ensembling removes the artifact, and recalibration passes the audit only by nullifying the claims: the head is uninformative, not merely miscalibrated. We release the toolkit and document two silent pitfalls that produced convincing but wrong audits of our own.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 17 of its 24 references that are in the map. First posted on arXiv 13 Jul 2026.

Builds on

17 of 24 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation29 Oct
  2. 2024Echoes of Socratic Doubt: Embracing Uncertainty in Calibrated Evidential Reinforcement Learning11 Feb

Other references · 15

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  3. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  4. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  5. 2017Visualizing and Understanding Atari Agentscited by 141
  6. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 70
  7. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  8. 2019Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learningcited by 60
  9. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  10. 2022Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learningcited by 27
  11. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 20
  12. 2021Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Drivingcited by 17
3 more
  1. 2021SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learningcited by 15
  2. 2023The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimationcited by 14
  3. 2023Diverse Projection Ensembles for Distributional Reinforcement Learningcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Assessing the Impact of Distribution Shift on Reinforcement Learning Performance
  2. 2021Conservative Offline Distributional Reinforcement Learning
  3. 2024A Distributional Analogue to the Successor Representation
  4. 2025Imitation Learning as Return Distribution Matching
  5. 2026Reinforcement Learning from Rich Feedback with Distributional DAgger
  6. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning
  7. 2024The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards
  8. 2025Distributional Inverse Reinforcement Learning