# Minimax Value Interval for Off-Policy Evaluation and Policy Optimization

arXiv: https://arxiv.org/abs/2002.02081 · HTML: https://mlsanity.com/paper/2002.02081/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2020.

## Built on by (most cited first, 11 of 11)

- [What are the Statistical Limits of Offline RL with Linear Function Approximation?](https://mlsanity.com/paper/2010.11895/) (2020)
- [Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency](https://mlsanity.com/paper/2102.02981/) (2021)
- [Universal Off-Policy Evaluation](https://mlsanity.com/paper/2104.12820/) (2021)
- [Instabilities of Offline RL with Pre-Trained Neural Representation](https://mlsanity.com/paper/2103.04947/) (2021)
- [Confidence-Conditioned Value Functions for Offline Reinforcement Learning](https://mlsanity.com/paper/2212.04607/) (2022)
- [Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Bounds](https://mlsanity.com/paper/2103.05741/) (2021)
- [On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction](https://mlsanity.com/paper/2106.00993/) (2021)
- [SOPE: Spectrum of Off-Policy Estimators](https://mlsanity.com/paper/2111.03936/) (2021)
- [Model-based Offline Reinforcement Learning with Local Misspecification](https://mlsanity.com/paper/2301.11426/) (2023)
- [Off-Policy Evaluation for Action-Dependent Non-Stationary Environments](https://mlsanity.com/paper/2301.10330/) (2023)
- [Operator Deep Q-Learning: Zero-Shot Reward Transferring in Reinforcement Learning](https://mlsanity.com/paper/2201.00236/) (2022)

## Abstract (arXiv)

We study minimax methods for off-policy evaluation (OPE) using value functions and marginalized importance weights. Despite that they hold promises of overcoming the exponential variance in traditional importance sampling, several key problems remain: (1) They require function approximation and are generally biased. For the sake of trustworthy OPE, is there anyway to quantify the biases? (2) They are split into two styles ("weight-learning" vs "value-learning"). Can we unify them? In this paper we answer both questions positively. By slightly altering the derivation of previous methods (one from each style; Uehara et al., 2020), we unify them into a single value interval that comes with a special type of double robustness: when either the value-function or the importance-weight class is well specified, the interval is valid and its length quantifies the misspecification of the other class. Our interval also provides a unified view of and new insights to some recent methods, and we further explore the implications of our results on exploration and exploitation in off-policy policy optimization with insufficient data coverage.