mlsanityData to 7 Oct 2026

Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation

arXiv 1910.07186 ↗First posted 16 Oct 2019cs.LG cs.AI stat.MLcited by 48

Not on the map. Why?

Abstract (arXiv)

Infinite horizon off-policy policy evaluation is a highly challenging task due to the excessively large variance of typical importance sampling (IS) estimators. Recently, Liu et al. (2018a) proposed an approach that significantly reduces the variance of infinite-horizon off-policy evaluation by estimating the stationary density ratio, but at the cost of introducing potentially high biases due to the error in density ratio estimation. In this paper, we develop a bias-reduced augmentation of their method, which can take advantage of a learned value function to obtain higher accuracy. Our method is doubly robust in that the bias vanishes when either the density ratio or the value function estimation is perfect. In general, when either of them is accurate, the bias can also be reduced. Both theoretical and empirical results show that our method yields significant advantages over previous methods.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2020: 2; 2021: 13; 2022: 17; 2023: 7; 2024: 5; 2025: 4.

48 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  3. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  4. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  5. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  6. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  7. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  8. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  9. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  10. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
  11. 2021Density Constrained Reinforcement Learningcited by 25
  12. 2022Markovian Interference in Experimentscited by 22
36 more
  1. 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 20
  2. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  3. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
  4. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  5. 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
  6. 2022Conformal Off-policy Predictioncited by 14
  7. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learningcited by 14
  8. 2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14
  9. 2021Continual Learning In Environments With Polynomial Mixing Timescited by 13
  10. 2024Offline Multi-task Transfer RL with Representational Penalizationcited by 13
  11. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
  12. 2021Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Boundscited by 11
  13. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  14. 2022Robust Imitation Learning from Corrupted Demonstrationscited by 10
  15. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learningcited by 10
  16. 2023Offline Learning of Closed-Loop Deep Brain Stimulation Controllers for Parkinson Disease Treatmentcited by 8
  17. 2024Optimizing Language Models for Human Preferences is a Causal Inference Problemcited by 8
  18. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  19. 2021Robust Generalization despite Distribution Shift via Minimum Discriminating Informationcited by 6
  20. 2021Stateful Offline Contextual Policy Evaluation and Learningcited by 6
  21. 2023Variational Latent Branching Model for Off-Policy Evaluationcited by 6
  22. 2023Off-Policy Evaluation for Human Feedbackcited by 6
  23. 2022A Reinforcement Learning Approach to Estimating Long-term Treatment Effectscited by 5
  24. 2023Marginalized Importance Sampling for Off-Environment Policy Evaluationcited by 5
  25. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  26. 2022A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Marketscited by 4
  27. 2021Pessimistic Model Selection for Offline Deep Reinforcement Learningcited by 3
  28. 2023A Reinforcement Learning Framework for Dynamic Mediation Analysiscited by 2
  29. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 2
  30. 2024Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferencescited by 2
  31. 2025Off-Policy Evaluation Under Nonignorable Missing Datacited by 2
  32. 2025Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Modelscited by 2
  33. 2021A Unified Off-Policy Evaluation Approach for General Value Functioncited by 1
  34. 2022Operator Deep Q-Learning: Zero-Shot Reward Transferring in Reinforcement Learningcited by 1
  35. 2022Biases in In Silico Evaluation of Molecular Optimization Methods and Bias-Reduced Evaluation Methodologycited by 1
  36. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1