mlsanityData to 7 Oct 2026

Importance Sampling Policy Evaluation with an Estimated Behavior Policy

arXiv 1806.01347 ↗First posted 4 Jun 2018cs.LG cs.AI stat.MLcited by 34

Not on the map. Why?

Abstract (arXiv)

We consider the problem of off-policy evaluation in Markov decision processes. Off-policy evaluation is the task of evaluating the expected return of one policy with data generated by a different, behavior policy. Importance sampling is a technique for off-policy evaluation that re-weights off-policy returns to account for differences in the likelihood of the returns between the two policies. In this paper, we study importance sampling with an estimated behavior policy where the behavior policy estimate comes from the same set of data used to compute the importance sampling estimate. We find that this estimator often lowers the mean squared error of off-policy evaluation compared to importance sampling with the true behavior policy or using a behavior policy that is estimated from a separate data set. Intuitively, estimating the behavior policy in this way corrects for error due to sampling in the action-space. Our empirical results also extend to other popular variants of importance sampling and show that estimating a non-Markovian behavior policy can further lower large-sample mean squared error even when the true behavior policy is Markovian.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 11; 2022: 4; 2023: 7; 2024: 6; 2025: 2; 2026: 3.

34 papers.

Sort:

By area:

  1. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  2. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  3. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  4. 2021Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learningcited by 53
  5. 2021Universal Off-Policy Evaluationcited by 43
  6. 2021Counterfactual Explanations in Sequential Decision Making Under Uncertaintycited by 36
  7. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  8. 2022Model-Free and Model-Based Policy Evaluation when Causality is Uncertaincited by 12
  9. 2025Data Retrieval with Importance Weights for Few-Shot Imitation Learningcited by 10
  10. 2022Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstractioncited by 7
  11. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  12. 2023Off-Policy Evaluation for Human Feedbackcited by 6
22 more
  1. 2021SOPE: Spectrum of Off-Policy Estimatorscited by 5
  2. 2023HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcarecited by 5
  3. 2024Bayesian Off-Policy Evaluation and Learning for Large Action Spacescited by 5
  4. 2024Reusing Historical Trajectories in Natural Policy Gradient via Importance Sampling: Convergence and Convergence Ratecited by 5
  5. 2021SGD with Coordinate Sampling: Theory and Practicecited by 4
  6. 2023Value-aware Importance Weighting for Off-policy Reinforcement Learningcited by 4
  7. 2023Offline Recommender System Evaluation under Unobserved Confoundingcited by 4
  8. 2024Primal-Dual Spectral Representation for Off-policy Evaluationcited by 4
  9. 2021Supervised Off-Policy Rankingcited by 3
  10. 2024Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policiescited by 3
  11. 2021Off Environment Evaluation Using Convex Risk Minimizationcited by 2
  12. 2022On the Reuse Bias in Off-Policy Reinforcement Learningcited by 2
  13. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 2
  14. 2026AIS: Adaptive Importance Sampling for Quantized RLcited by 2
  15. 2021Case-based off-policy policy evaluation using prototype learningcited by 1
  16. 2022A cGAN Ensemble-based Uncertainty-aware Surrogate Model for Offline Model-based Optimization in Industrial Control Problemscited by 1
  17. 2024Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluationcited by 1
  18. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  19. 2023Minimax Weight Learning for Absorbing MDPscited by 0
  20. 2023Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learningcited by 0
  21. 2026Deep Doubly Debiased Longitudinal Effect Estimation with ICE G-Computationcited by 0
  22. 2026Smooth Multi-Policy Causal Effect Estimation in Longitudinal Settingscited by 0