mlsanityData to 7 Oct 2026

CoinDICE: Off-Policy Confidence Interval Estimation

arXiv 2010.11652 ↗First posted 22 Oct 2020cs.LG stat.MLcited by 64

Not on the map. Why?

Abstract (arXiv)

We study high-confidence behavior-agnostic off-policy evaluation in reinforcement learning, where the goal is to estimate a confidence interval on a target policy's value, given only access to a static experience dataset collected by unknown behavior policies. Starting from a function space embedding of the linear program formulation of the Q-function, we obtain an optimization problem with generalized estimating equation constraints. By applying the generalized empirical likelihood method to the resulting Lagrangian, we propose CoinDICE, a novel and efficient algorithm for computing confidence intervals. Theoretically, we prove the obtained confidence intervals are valid, in both asymptotic and finite-sample regimes. Empirically, we show in a variety of benchmarks that the confidence interval estimates are tighter and more accurate than existing methods.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 15; 2023: 19; 2024: 8; 2025: 6; 2026: 4.

64 papers.

Sort:

By area:

  1. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  2. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  3. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  4. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  5. 2021Universal Off-Policy Evaluationcited by 43
  6. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  7. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  8. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  9. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  10. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  11. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  12. 2021On the Optimality of Batch Policy Optimization Algorithmscited by 28
38 more
  1. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  2. 2021Online Bootstrap Inference For Policy Evaluation in Reinforcement Learningcited by 26
  3. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
  4. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learningcited by 20
  5. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  6. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  7. 2024Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learningcited by 16
  8. 2021Off-policy Confidence Sequencescited by 15
  9. 2022Conformal Off-policy Predictioncited by 14
  10. 2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14
  11. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learningcited by 13
  12. 2021Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Boundscited by 11
  13. 2022Robust Imitation Learning from Corrupted Demonstrationscited by 10
  14. 2023Conservative State Value Estimation for Offline Reinforcement Learningcited by 9
  15. 2023Offline Learning of Closed-Loop Deep Brain Stimulation Controllers for Parkinson Disease Treatmentcited by 8
  16. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  17. 2023Variational Latent Branching Model for Off-Policy Evaluationcited by 6
  18. 2023Sequential Counterfactual Risk Minimizationcited by 6
  19. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  20. 2023Off-Policy Evaluation for Human Feedbackcited by 6
  21. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  22. 2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
  23. 2022A Reinforcement Learning Approach to Estimating Long-term Treatment Effectscited by 5
  24. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  25. 2025Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approachcited by 5
  26. 2022Generalizing Off-Policy Evaluation From a Causal Perspective For Sequential Decision-Makingcited by 4
  27. 2023Offline Diversity Maximization Under Imitation Constraintscited by 4
  28. 2023Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learningcited by 4
  29. 2023Online Estimation and Inference for Robust Policy Evaluation in Reinforcement Learningcited by 4
  30. 2024Distributionally Robust Policy Evaluation under General Covariate Shift in Contextual Banditscited by 4
  31. 2024Primal-Dual Spectral Representation for Off-policy Evaluationcited by 4
  32. 2021The f-Divergence Reinforcement Learning Frameworkcited by 3
  33. 2023Uncertainty-Aware Instance Reweighting for Off-Policy Learningcited by 3
  34. 2023Understanding Deep Generative Models with Generalized Empirical Likelihoodscited by 3
  35. 2023Off-Policy Evaluation for Action-Dependent Non-Stationary Environmentscited by 2
  36. 2023A Reinforcement Learning Framework for Dynamic Mediation Analysiscited by 2
  37. 2024Multiple-policy Evaluation via Density Estimationcited by 2
  38. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 2

50 most cited of 64 All 64 (JSON)