# Representation Balancing MDPs for Off-Policy Policy Evaluation

arXiv: https://arxiv.org/abs/1805.09044 · HTML: https://mlsanity.com/paper/1805.09044/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2018.

## Built on by (most cited first, 20 of 20)

- [Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning](https://mlsanity.com/paper/2108.08812/) (2021)
- [Average-Reward Off-Policy Policy Evaluation with Function Approximation](https://mlsanity.com/paper/2101.02808/) (2021)
- [Data-Driven Offline Decision-Making via Invariant Representation Learning](https://mlsanity.com/paper/2211.11349/) (2022)
- [Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithm](https://mlsanity.com/paper/2102.09318/) (2021)
- [Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximation](https://mlsanity.com/paper/2105.12540/) (2021)
- [A Unified Framework for Alternating Offline Model Training and Policy Learning](https://mlsanity.com/paper/2210.05922/) (2022)
- [Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Bounds](https://mlsanity.com/paper/2103.05741/) (2021)
- [Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstraction](https://mlsanity.com/paper/2212.07486/) (2022)
- [Off-policy Evaluation in Doubly Inhomogeneous Environments](https://mlsanity.com/paper/2306.08719/) (2023)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluation](https://mlsanity.com/paper/2111.14272/) (2021)
- [Model-based Offline Reinforcement Learning with Local Misspecification](https://mlsanity.com/paper/2301.11426/) (2023)
- [Leveraging Factored Action Spaces for Off-Policy Evaluation](https://mlsanity.com/paper/2307.07014/) (2023)
- [OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators](https://mlsanity.com/paper/2405.17708/) (2024)
- [Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation](https://mlsanity.com/paper/2410.02172/) (2024)
- [Model Selection for Production System via Automated Online Experiments](https://mlsanity.com/paper/2105.13420/) (2021)
- [Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learning](https://mlsanity.com/paper/2106.00589/) (2021)
- [K-Nearest-Neighbor Resampling for Off-Policy Evaluation in Stochastic Control](https://mlsanity.com/paper/2306.04836/) (2023)
- [Concept-driven Off Policy Evaluation](https://mlsanity.com/paper/2411.19395/) (2024)
- [Short-Long Policy Evaluation with Novel Actions](https://mlsanity.com/paper/2407.03674/) (2024)

## Abstract (arXiv)

We study the problem of off-policy policy evaluation (OPPE) in RL. In contrast to prior work, we consider how to estimate both the individual policy value and average policy value accurately. We draw inspiration from recent work in causal reasoning, and propose a new finite sample generalization error bound for value estimates from MDP models. Using this upper bound as an objective, we develop a learning algorithm of an MDP model with a balanced representation, and show that our approach can yield substantially lower MSE in common synthetic benchmarks and a HIV treatment simulation domain.