# State Relevance for Off-Policy Evaluation

arXiv: https://arxiv.org/abs/2109.06310 · HTML: https://mlsanity.com/paper/2109.06310/

## Key facts

- Cited by 3 papers in the map (data through 7 Oct 2026).
- Builds on 9 of its 26 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 9 of 9)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation](https://mlsanity.com/paper/1810.12429/) (2018)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling](https://mlsanity.com/paper/1906.03393/) (2019)
- [Doubly robust off-policy evaluation with shrinkage](https://mlsanity.com/paper/1907.09623/) (2019)
- [Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions](https://mlsanity.com/paper/2002.03478/) (2020)
- [Conditional Importance Sampling for Off-Policy Learning](https://mlsanity.com/paper/1910.07479/) (2019)

## Built on by (most cited first, 3 of 3)

- [Concept-driven Off Policy Evaluation](https://mlsanity.com/paper/2411.19395/) (2024)
- [Formulating Reinforcement Learning for Human-Robot Collaboration through Off-Policy Evaluation](https://mlsanity.com/paper/2602.02530/) (2026)
- [Smooth Multi-Policy Causal Effect Estimation in Longitudinal Settings](https://mlsanity.com/paper/2605.14284/) (2026)

## Abstract (arXiv)

Importance sampling-based estimators for off-policy evaluation (OPE) are valued for their simplicity, unbiasedness, and reliance on relatively few assumptions. However, the variance of these estimators is often high, especially when trajectories are of different lengths. In this work, we introduce Omitting-States-Irrelevant-to-Return Importance Sampling (OSIRIS), an estimator which reduces variance by strategically omitting likelihood ratios associated with certain states. We formalize the conditions under which OSIRIS is unbiased and has lower variance than ordinary importance sampling, and we demonstrate these properties empirically.