Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design

Online Monte Carlo evaluation is a fundamental tool for assessing policy performance in reinforcement learning and sequential decision-making problems arising in operations research. However, achieving accurate estimates often requires extensive online interaction with the environment, which can be costly or impractical in many real-world settings. In this paper, we develop a framework that improves the sample efficiency of online Monte Carlo estimators while preserving unbiasedness. We first derive a closed-form optimal behavior policy that minimizes estimator variance under unbiasedness constraints. We then propose practical algorithms for learning the proposed behavior policy from previously collected offline data, enabling improved online evaluation without requiring estimation of the environment transition model. We provide theoretical analysis that quantifies the resulting variance reduction and analyzes the impact of approximation errors. Empirical studies across diverse environments demonstrate substantial improvements in online sample efficiency compared with standard on-policy Monte Carlo evaluation and existing baseline methods. Our results provide a unified framework for optimal behavior policy design in off-policy evaluation, with applications to reinforcement learning and operations research.