# Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation

arXiv: https://arxiv.org/abs/2510.26026 · HTML: https://mlsanity.com/paper/2510.26026/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 28 of 28)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [A Theoretical Analysis of Deep Q-Learning](https://mlsanity.com/paper/1901.00137/) (2019)
- [Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach](https://mlsanity.com/paper/1506.02188/) (2015)
- [Conformal Inference of Counterfactuals and Individual Treatment Effects](https://mlsanity.com/paper/2006.06138/) (2020)
- [Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes](https://mlsanity.com/paper/1908.08526/) (2019)
- [Fully Parameterized Quantile Function for Distributional Reinforcement Learning](https://mlsanity.com/paper/1911.02140/) (2019)
- [Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settings](https://mlsanity.com/paper/2001.04515/) (2020)
- [An Analysis of Quantile Temporal-Difference Learning](https://mlsanity.com/paper/2301.04462/) (2023)
- [Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping](https://mlsanity.com/paper/1910.02787/) (2019)
- [Prediction Sets Adaptive to Unknown Covariate Shift](https://mlsanity.com/paper/2203.06126/) (2022)
- [Multi Split Conformal Prediction](https://mlsanity.com/paper/2103.00627/) (2021)
- [Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structures](https://mlsanity.com/paper/1808.09011/) (2018)
- [Wasserstein-regularized Conformal Prediction under General Distribution Shift](https://mlsanity.com/paper/2501.13430/) (2025)
- [Conformal Off-policy Prediction](https://mlsanity.com/paper/2206.06711/) (2022)
- [A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDP](https://mlsanity.com/paper/2202.04157/) (2022)
- [Conformal Prediction Intervals for Markov Decision Process Trajectories](https://mlsanity.com/paper/2206.04860/) (2022)
- [Safe POMDP Online Planning among Dynamic Agents via Adaptive Conformal Prediction](https://mlsanity.com/paper/2404.15557/) (2024)
- [Conformal Off-Policy Evaluation in Markov Decision Processes](https://mlsanity.com/paper/2304.02574/) (2023)
- [Conformal Predictions under Markovian Data](https://mlsanity.com/paper/2407.15277/) (2024)
- [Computationally and Sample Efficient Safe Reinforcement Learning Using Adaptive Conformal Prediction](https://mlsanity.com/paper/2503.17678/) (2025)
- [Distributional Off-policy Evaluation with Bellman Residual Minimization](https://mlsanity.com/paper/2402.01900/) (2024)

## Built on by (most cited first, 2 of 2)

- [On Optimal Data Splitting for Split Conformal Prediction](https://mlsanity.com/paper/2606.31600/) (2026)
- [Auditing the Risk Claims of Distributional Reinforcement Learning](https://mlsanity.com/paper/2607.11607/) (2026)

## Abstract (arXiv)

Reliable uncertainty quantification is crucial for reinforcement learning (RL) in high-stakes settings. We propose a unified conformal prediction framework for infinite-horizon policy evaluation that constructs distribution-free prediction intervals {for returns} in both on-policy and off-policy settings. Our method integrates distributional RL with conformal calibration, addressing challenges such as unobserved returns, temporal dependencies, and distributional shifts. We propose a modular pseudo-return construction based on truncated rollouts and a time-aware calibration strategy using experience replay and weighted subsampling. These innovations mitigate model bias and restore approximate exchangeability, enabling uncertainty quantification even under policy shifts. Our theoretical analysis provides coverage guarantees that account for model misspecification and importance weight estimation. Empirical results, including experiments in synthetic and benchmark environments like Mountain Car, show that our method significantly improves coverage and reliability over standard distributional RL baselines.