mlsanityData to 14 Oct 2026

Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation

arXiv 2510.26026 ↗First posted 29 Oct 2025stat.ML cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Reliable uncertainty quantification is crucial for reinforcement learning (RL) in high-stakes settings. We propose a unified conformal prediction framework for infinite-horizon policy evaluation that constructs distribution-free prediction intervals {for returns} in both on-policy and off-policy settings. Our method integrates distributional RL with conformal calibration, addressing challenges such as unobserved returns, temporal dependencies, and distributional shifts. We propose a modular pseudo-return construction based on truncated rollouts and a time-aware calibration strategy using experience replay and weighted subsampling. These innovations mitigate model bias and restore approximate exchangeability, enabling uncertainty quantification even under policy shifts. Our theoretical analysis provides coverage guarantees that account for model misspecification and importance weight estimation. Empirical results, including experiments in synthetic and benchmark environments like Mountain Car, show that our method significantly improves coverage and reliability over standard distributional RL baselines.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 45 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

28 of 45 references.

References · 28

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2015Prioritized Experience Replaycited by 1,099
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  4. 2016Distribution-Free Predictive Inference For Regressioncited by 668
  5. 2019Conformalized Quantile Regressioncited by 629
  6. 2019Conformal Prediction Under Covariate Shiftcited by 519
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  8. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  9. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  10. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  11. 2020Conformal Inference of Counterfactuals and Individual Treatment Effectscited by 151
  12. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
16 more
  1. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  2. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  3. 2023An Analysis of Quantile Temporal-Difference Learningcited by 31
  4. 2019Quantile QT-Opt for Risk-Aware Vision-Based Robotic Graspingcited by 29
  5. 2022Prediction Sets Adaptive to Unknown Covariate Shiftcited by 22
  6. 2021Multi Split Conformal Predictioncited by 21
  7. 2018Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structurescited by 20
  8. 2025Wasserstein-regularized Conformal Prediction under General Distribution Shiftcited by 15
  9. 2022Conformal Off-policy Predictioncited by 14
  10. 2022A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDPcited by 13
  11. 2022Conformal Prediction Intervals for Markov Decision Process Trajectoriescited by 12
  12. 2024Safe POMDP Online Planning among Dynamic Agents via Adaptive Conformal Predictioncited by 8
  13. 2023Conformal Off-Policy Evaluation in Markov Decision Processescited by 6
  14. 2024Conformal Predictions under Markovian Datacited by 4
  15. 2025Computationally and Sample Efficient Safe Reinforcement Learning Using Adaptive Conformal Predictioncited by 4
  16. 2024Distributional Off-policy Evaluation with Bellman Residual Minimizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026On Optimal Data Splitting for Split Conformal Predictioncited by 1
  2. 2026Auditing the Risk Claims of Distributional Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026UNIQ: Conformal Calibration for Adaptive Conservatism in Offline Reinforcement Learning
  2. 2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning
  3. 2023Selective Uncertainty Propagation in Offline RL
  4. 2026Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning
  5. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
  6. 2020Reliable Off-policy Evaluation for Reinforcement Learning
  7. 2026Offline Policy Optimization with Posterior Sampling
  8. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning