Distributional Off-Policy Evaluation with Deep Quantile Process Regression
Not on the map. Why?
Abstract (arXiv)
This paper investigates the off-policy evaluation (OPE) problem from a distributional perspective. Rather than focusing solely on the expectation of the total return, as in most existing OPE methods, we aim to estimate the entire return distribution. To this end, we introduce a quantile-based approach for OPE using deep quantile process regression, presenting a novel algorithm called Deep Quantile Process regression-based Off-Policy Evaluation (DQPOPE). We provide new theoretical insights into the deep quantile process regression technique, extending existing approaches that estimate discrete quantiles to estimate a continuous quantile function. A key contribution of our work is the rigorous sample complexity analysis for distributional OPE with deep neural networks, bridging theoretical analysis with practical algorithmic implementations. We show that DQPOPE achieves statistical advantages by estimating the full return distribution using the same sample size required to estimate a single policy value using conventional methods. Empirical studies further show that DQPOPE provides significantly more precise and robust policy value estimates than standard methods, thereby enhancing the practical applicability and effectiveness of distributional reinforcement learning approaches.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 38 references that are in the map. First posted on arXiv 20 Apr 2026.
Builds on
18 of 38 references.
References · 18
- 2017A Distributional Perspective on Reinforcement Learningcited by 765
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
- 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
- 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
- 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
- 2019A Theoretical Analysis of Deep Q-Learningcited by 215
- 2019Wasserstein Robust Reinforcement Learningcited by 54
- 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
- 2023An Analysis of Quantile Temporal-Difference Learningcited by 31
- 2019Quantile QT-Opt for Risk-Aware Vision-Based Robotic Graspingcited by 29
- 2020Quantile regression with deep ReLU Networks: Estimators and minimax ratescited by 24
- 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 20
6 more
- 2023The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimationcited by 14
- 2022Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networkscited by 12
- 2017Distributed inference for quantile regression processescited by 11
- 2021Sample Complexity of Offline Reinforcement Learning with Deep ReLU Networkscited by 8
- 2023Estimation and Inference in Distributional Reinforcement Learningcited by 8
- 2024Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Modelcited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Statistical Efficiency and Inference of Quantile Distributional Reinforcement Learning
- 2026Distorted Distributional Policy Evaluation for Offline Reinforcement Learning
- 2023Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networks
- 2026Quantile Geometry Regularization for Distributional Reinforcement Learning
- 2024Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation
- 2024A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning
- 2025A Principled Path to Fitted Distributional Evaluation
- 2025A Differential Perspective on Distributional Reinforcement Learning