# CoinDICE: Off-Policy Confidence Interval Estimation

arXiv: https://arxiv.org/abs/2010.11652 · HTML: https://mlsanity.com/paper/2010.11652/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2020.

## Built on by (most cited first, 30 of 64)

- [OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2106.10783/) (2021)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters](https://mlsanity.com/paper/2205.13703/) (2022)
- [COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2204.08957/) (2022)
- [Universal Off-Policy Evaluation](https://mlsanity.com/paper/2104.12820/) (2021)
- [Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching](https://mlsanity.com/paper/2202.02433/) (2022)
- [Bootstrapping Fitted Q-Evaluation for Off-Policy Inference](https://mlsanity.com/paper/2102.03607/) (2021)
- [Deeply-Debiased Off-Policy Interval Estimation](https://mlsanity.com/paper/2105.04646/) (2021)
- [A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes](https://mlsanity.com/paper/2111.06784/) (2021)
- [Variance-Aware Off-Policy Evaluation with Linear Function Approximation](https://mlsanity.com/paper/2106.11960/) (2021)
- [Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process](https://mlsanity.com/paper/2202.10589/) (2022)
- [On the Optimality of Batch Policy Optimization Algorithms](https://mlsanity.com/paper/2104.02293/) (2021)
- [Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptotics](https://mlsanity.com/paper/2105.03863/) (2021)
- [Online Bootstrap Inference For Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2108.03706/) (2021)
- [LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2202.13536/) (2022)
- [Confidence-Conditioned Value Functions for Offline Reinforcement Learning](https://mlsanity.com/paper/2212.04607/) (2022)
- [An Instrumental Variable Approach to Confounded Off-Policy Evaluation](https://mlsanity.com/paper/2212.14468/) (2022)
- [Survival Instinct in Offline Reinforcement Learning](https://mlsanity.com/paper/2306.03286/) (2023)
- [Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning](https://mlsanity.com/paper/2405.14335/) (2024)
- [Off-policy Confidence Sequences](https://mlsanity.com/paper/2102.09540/) (2021)
- [Conformal Off-policy Prediction](https://mlsanity.com/paper/2206.06711/) (2022)
- [A Unified Framework for Alternating Offline Model Training and Policy Learning](https://mlsanity.com/paper/2210.05922/) (2022)
- [Oracle Inequalities for Model Selection in Offline Reinforcement Learning](https://mlsanity.com/paper/2211.02016/) (2022)
- [Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Bounds](https://mlsanity.com/paper/2103.05741/) (2021)
- [Robust Imitation Learning from Corrupted Demonstrations](https://mlsanity.com/paper/2201.12594/) (2022)
- [Conservative State Value Estimation for Offline Reinforcement Learning](https://mlsanity.com/paper/2302.06884/) (2023)
- [Offline Learning of Closed-Loop Deep Brain Stimulation Controllers for Parkinson Disease Treatment](https://mlsanity.com/paper/2302.02477/) (2023)
- [Robust Offline Reinforcement learning with Heavy-Tailed Rewards](https://mlsanity.com/paper/2310.18715/) (2023)
- [Variational Latent Branching Model for Off-Policy Evaluation](https://mlsanity.com/paper/2301.12056/) (2023)
- [Sequential Counterfactual Risk Minimization](https://mlsanity.com/paper/2302.12120/) (2023)

## Abstract (arXiv)

We study high-confidence behavior-agnostic off-policy evaluation in reinforcement learning, where the goal is to estimate a confidence interval on a target policy's value, given only access to a static experience dataset collected by unknown behavior policies. Starting from a function space embedding of the linear program formulation of the Q-function, we obtain an optimization problem with generalized estimating equation constraints. By applying the generalized empirical likelihood method to the resulting Lagrangian, we propose CoinDICE, a novel and efficient algorithm for computing confidence intervals. Theoretically, we prove the obtained confidence intervals are valid, in both asymptotic and finite-sample regimes. Empirically, we show in a variety of benchmarks that the confidence interval estimates are tighter and more accurate than existing methods.