# Balanced off-policy evaluation in general action spaces

arXiv: https://arxiv.org/abs/1906.03694 · HTML: https://mlsanity.com/paper/1906.03694/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2019.

## Built on by (most cited first, 11 of 11)

- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Off-Policy Evaluation for Large Action Spaces via Embeddings](https://mlsanity.com/paper/2202.06317/) (2022)
- [Marginal Density Ratio for Off-Policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/2312.01457/) (2023)
- [Learning Complementary Policies for Human-AI Teams](https://mlsanity.com/paper/2302.02944/) (2023)
- [Loss Functions for Discrete Contextual Pricing with Observational Data](https://mlsanity.com/paper/2111.09933/) (2021)
- [Balanced Off-Policy Evaluation for Personalized Pricing](https://mlsanity.com/paper/2302.12736/) (2023)
- [Confounding-Robust Policy Improvement with Human-AI Teams](https://mlsanity.com/paper/2310.08824/) (2023)
- [Off-Policy Evaluation in Embedded Spaces](https://mlsanity.com/paper/2203.02807/) (2022)
- [Policy Learning for Optimal Individualized Dose Intervals](https://mlsanity.com/paper/2202.12234/) (2022)
- [IntOPE: Off-Policy Evaluation in the Presence of Interference](https://mlsanity.com/paper/2408.13484/) (2024)
- [Learning density ratios in causal inference using Bregman-Riesz regression](https://mlsanity.com/paper/2510.16127/) (2025)

## Abstract (arXiv)

Estimation of importance sampling weights for off-policy evaluation of contextual bandits often results in imbalance - a mismatch between the desired and the actual distribution of state-action pairs after weighting. In this work we present balanced off-policy evaluation (B-OPE), a generic method for estimating weights which minimize this imbalance. Estimation of these weights reduces to a binary classification problem regardless of action type. We show that minimizing the risk of the classifier implies minimization of imbalance to the desired counterfactual distribution of state-action pairs. The classifier loss is tied to the error of the off-policy estimate, allowing for easy tuning of hyperparameters. We provide experimental evidence that B-OPE improves weighting-based approaches for offline policy evaluation in both discrete and continuous action spaces.