# Data-Efficient Policy Evaluation Through Behavior Policy Search

arXiv: https://arxiv.org/abs/1706.03469 · HTML: https://mlsanity.com/paper/1706.03469/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2017.

## Built on by (most cited first, 26 of 26)

- [ReVar: Strengthening Policy Evaluation via Reduced Variance Sampling](https://mlsanity.com/paper/2203.04510/) (2022)
- [Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2111.14552/) (2021)
- [Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design](https://mlsanity.com/paper/2301.13734/) (2023)
- [Policy Gradient with Active Importance Sampling](https://mlsanity.com/paper/2405.05630/) (2024)
- [Doubly Optimal Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/2410.02226/) (2024)
- [Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments](https://mlsanity.com/paper/2403.17285/) (2024)
- [Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning](https://mlsanity.com/paper/2410.05655/) (2024)
- [Adaptive Exploration for Multi-Reward Multi-Policy Evaluation](https://mlsanity.com/paper/2502.02516/) (2025)
- [Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approach](https://mlsanity.com/paper/2505.20130/) (2025)
- [Logarithmic Neyman Regret for Adaptive Estimation of the Average Treatment Effect](https://mlsanity.com/paper/2411.14341/) (2024)
- [SPEED: Experimental Design for Policy Evaluation in Linear Heteroscedastic Bandits](https://mlsanity.com/paper/2301.12357/) (2023)
- [SaVeR: Optimal Data Collection Strategy for Safe Policy Evaluation in Tabular MDP](https://mlsanity.com/paper/2406.02165/) (2024)
- [Adaptive Exploration for Data-Efficient General Value Function Evaluations](https://mlsanity.com/paper/2405.07838/) (2024)
- [Optimistic Algorithms for Adaptive Estimation of the Average Treatment Effect](https://mlsanity.com/paper/2502.04673/) (2025)
- [Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2511.10843/) (2025)
- [Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluation](https://mlsanity.com/paper/2608.24146/) (2026)
- [Efficient Open-world Reinforcement Learning via Knowledge Distillation and Autonomous Rule Discovery](https://mlsanity.com/paper/2311.14270/) (2023)
- [Truncating Trajectories in Monte Carlo Policy Evaluation: an Adaptive Approach](https://mlsanity.com/paper/2410.13463/) (2024)
- [Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation](https://mlsanity.com/paper/2505.22492/) (2025)
- [Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learning](https://mlsanity.com/paper/2602.01853/) (2026)
- [Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning](https://mlsanity.com/paper/2602.12375/) (2026)
- [Clustered KL-barycenter design for policy evaluation](https://mlsanity.com/paper/2503.02735/) (2025)
- [Evaluation-Aware Reinforcement Learning](https://mlsanity.com/paper/2509.19464/) (2025)
- [Actor-Critic with Active Importance Sampling](https://mlsanity.com/paper/2605.07094/) (2026)
- [Training ML Models with Predictable Failures](https://mlsanity.com/paper/2605.15134/) (2026)
- [agent-risk: Risk Assessment for Changes to AI Agents](https://mlsanity.com/paper/2610.16280/) (2026)

## Abstract (arXiv)

We consider the task of evaluating a policy for a Markov decision process (MDP). The standard unbiased technique for evaluating a policy is to deploy the policy and observe its performance. We show that the data collected from deploying a different policy, commonly called the behavior policy, can be used to produce unbiased estimates with lower mean squared error than this standard technique. We derive an analytic expression for the optimal behavior policy --- the behavior policy that minimizes the mean squared error of the resulting estimates. Because this expression depends on terms that are unknown in practice, we propose a novel policy evaluation sub-problem, behavior policy search: searching for a behavior policy that reduces mean squared error. We present a behavior policy search algorithm and empirically demonstrate its effectiveness in lowering the mean squared error of policy performance estimates.