# More Efficient Off-Policy Evaluation through Regularized Targeted Learning

arXiv: https://arxiv.org/abs/1912.06292 · HTML: https://mlsanity.com/paper/1912.06292/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2019.

## Built on by (most cited first, 17 of 17)

- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Future-Dependent Value-Based Off-Policy Evaluation in POMDPs](https://mlsanity.com/paper/2207.13081/) (2022)
- [Control Variates for Slate Off-Policy Evaluation](https://mlsanity.com/paper/2106.07914/) (2021)
- [Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders](https://mlsanity.com/paper/2302.00662/) (2023)
- [Doubly Robust Alignment for Large Language Models](https://mlsanity.com/paper/2506.01183/) (2025)
- [Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning](https://mlsanity.com/paper/2006.03886/) (2020)
- [Treatment Effect Estimation for Optimal Decision-Making](https://mlsanity.com/paper/2505.13092/) (2025)
- [Stateful Offline Contextual Policy Evaluation and Learning](https://mlsanity.com/paper/2110.10081/) (2021)
- [Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments](https://mlsanity.com/paper/2403.17285/) (2024)
- [Efficient and Sharp Off-Policy Learning under Unobserved Confounding](https://mlsanity.com/paper/2502.13022/) (2025)
- [Debiasing Samples from Online Learning Using Bootstrap](https://mlsanity.com/paper/2108.00236/) (2021)
- [Data-Driven Influence Functions for Optimization-Based Causal Inference](https://mlsanity.com/paper/2208.13701/) (2022)
- [Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation](https://mlsanity.com/paper/2505.22492/) (2025)
- [Adaptive Doubly Robust Estimator from Non-stationary Logging Policy under a Convergence of Average Probability](https://mlsanity.com/paper/2102.08975/) (2021)
- [Deep Doubly Debiased Longitudinal Effect Estimation with ICE G-Computation](https://mlsanity.com/paper/2602.12379/) (2026)
- [Smooth Multi-Policy Causal Effect Estimation in Longitudinal Settings](https://mlsanity.com/paper/2605.14284/) (2026)
- [From Unsupervised Subgroups to Hypothetical State-Intervention Policies: An Evaluation of Selected Subgrouping Methods in Observational Health Data](https://mlsanity.com/paper/2607.26521/) (2026)

## Abstract (arXiv)

We study the problem of off-policy evaluation (OPE) in Reinforcement Learning (RL), where the aim is to estimate the performance of a new policy given historical data that may have been generated by a different policy, or policies. In particular, we introduce a novel doubly-robust estimator for the OPE problem in RL, based on the Targeted Maximum Likelihood Estimation principle from the statistical causal inference literature. We also introduce several variance reduction techniques that lead to impressive performance gains in off-policy evaluation. We show empirically that our estimator uniformly wins over existing off-policy evaluation methods across multiple RL environments and various levels of model misspecification. Finally, we further the existing theoretical analysis of estimators for the RL off-policy estimation problem by showing their OP(1/√n) rate of convergence and characterizing their asymptotic distribution.