mlsanityData to 7 Oct 2026

Distributional Offline Policy Evaluation with Predictive Error Guarantees

arXiv 2302.09456 ↗First posted 19 Feb 2023cs.LGcited by 19

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

We study the problem of estimating the distribution of the return of a policy using an offline dataset that is not generated from the policy, i.e., distributional offline policy evaluation (OPE). We propose an algorithm called Fitted Likelihood Estimation (FLE), which conducts a sequence of Maximum Likelihood Estimation (MLE) and has the flexibility of integrating any state-of-the-art probabilistic generative models as long as it can be trained via MLE. FLE can be used for both finite-horizon and infinite-horizon discounted settings where rewards can be multi-dimensional vectors. Our theoretical results show that for both finite-horizon and infinite-horizon discounted settings, FLE can learn distributions that are close to the ground truth under total variation distance and Wasserstein distance, respectively. Our theoretical results hold under the conditions that the offline data covers the test policy's traces and that the supervised learning MLE procedures succeed. Experimentally, we demonstrate the performance of FLE with two generative models, Gaussian mixture models and diffusion models. For the multi-dimensional reward setting, FLE with diffusion models is capable of estimating the complicated distribution of the return of a test policy.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 31 of its 51 references that are in the map. First posted on arXiv 19 Feb 2023.

Builds on

31 of 51 references.

References · 31

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  4. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  6. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  8. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  9. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  10. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  11. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  12. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
19 more
  1. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 91
  2. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  3. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  4. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  5. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  6. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  7. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  8. 2021Universal Off-Policy Evaluationcited by 43
  9. 2022PAC Reinforcement Learning for Predictive State Representationscited by 42
  10. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  11. 2023An Analysis of Quantile Temporal-Difference Learningcited by 30
  12. 2021Off-Policy Risk Assessment in Contextual Banditscited by 29
  13. 2021Distributional Reinforcement Learning for Multi-Dimensional Reward Functionscited by 21
  14. 2019A Wasserstein distance approach for concentration of empirical risk estimatescited by 18
  15. 2022Learning Bellman Complete Representations for Offline Policy Evaluationcited by 17
  16. 2018Distributional Multivariate Policy Evaluation and Exploration with the Bellman GANcited by 16
  17. 2018GAN Q-learningcited by 7
  18. 2022Off-Policy Risk Assessment in Markov Decision Processescited by 7
  19. 2021Bayesian Distributional Policy Gradientscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 5; 2025: 4; 2026: 5.

19 papers.

Sort:

By area:

  1. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  2. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  3. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learningcited by 23
  4. 2024Foundations of Multivariate Distributional Reinforcement Learningcited by 16
  5. 2023Estimation and Inference in Distributional Reinforcement Learningcited by 8
  6. 2024The Central Role of the Loss Function in Reinforcement Learningcited by 8
  7. 2024Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Modelcited by 7
  8. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6
  9. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 5
  10. 2023Distributional Bellman Operators over Mean Embeddingscited by 4
  11. 2026Compositional Planning with Jumpy World Modelscited by 4
  12. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximationcited by 3
7 more
  1. 2025Doubly-Robust Estimation of Counterfactual Policy Mean Embeddingscited by 3
  2. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learningcited by 2
  3. 2026Quotient-Categorical Representations for Bellman-Compatible Average-Reward Distributional Reinforcement Learningcited by 1
  4. 2025Accelerated Distributional Temporal Difference Learning with Linear Function Approximationcited by 0
  5. 2026Distributional Off-Policy Evaluation with Deep Quantile Process Regressioncited by 0
  6. 2026Auditing the Risk Claims of Distributional Reinforcement Learningcited by 0
  7. 2026Tail-Influence Sampling for CVaR Policy Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
  2. 2026Generative OOD-regularized Model-based Policy Optimization
  3. 2024Learning from Sparse Offline Datasets via Conservative Density Estimation
  4. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
  5. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows
  6. 2024OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators
  7. 2023Offline Reinforcement Learning with Additional Covering Distributions
  8. 2026Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Random