# Combining Parametric and Nonparametric Models for Off-Policy Evaluation

arXiv: https://arxiv.org/abs/1905.05787 · HTML: https://mlsanity.com/paper/1905.05787/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2019.

## Built on by (most cited first, 17 of 17)

- [Average-Reward Off-Policy Policy Evaluation with Function Approximation](https://mlsanity.com/paper/2101.02808/) (2021)
- [Learning-to-defer for sequential medical decision-making under uncertainty](https://mlsanity.com/paper/2109.06312/) (2021)
- [Model-Free and Model-Based Policy Evaluation when Causality is Uncertain](https://mlsanity.com/paper/2204.00956/) (2022)
- [When is Agnostic Reinforcement Learning Statistically Tractable?](https://mlsanity.com/paper/2310.06113/) (2023)
- [An Optimal Tightness Bound for the Simulation Lemma](https://mlsanity.com/paper/2406.16249/) (2024)
- [Off-policy Evaluation in Doubly Inhomogeneous Environments](https://mlsanity.com/paper/2306.08719/) (2023)
- [Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments](https://mlsanity.com/paper/2403.17285/) (2024)
- [Generalizing Off-Policy Evaluation From a Causal Perspective For Sequential Decision-Making](https://mlsanity.com/paper/2201.08262/) (2022)
- [OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators](https://mlsanity.com/paper/2405.17708/) (2024)
- [Coarse-Grained Smoothness for RL in Metric Spaces](https://mlsanity.com/paper/2110.12276/) (2021)
- [K-Nearest-Neighbor Resampling for Off-Policy Evaluation in Stochastic Control](https://mlsanity.com/paper/2306.04836/) (2023)
- [Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferences](https://mlsanity.com/paper/2407.17910/) (2024)
- [Decision-Point Guided Safe Policy Improvement](https://mlsanity.com/paper/2410.09361/) (2024)
- [Limit Order Book Simulation and Trade Evaluation with K-Nearest-Neighbor Resampling](https://mlsanity.com/paper/2409.06514/) (2024)
- [Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation](https://mlsanity.com/paper/2505.22492/) (2025)
- [Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning](https://mlsanity.com/paper/2308.14897/) (2023)
- [When Machine Learning Meets Importance Sampling: A More Efficient Rare Event Estimation Approach](https://mlsanity.com/paper/2504.13982/) (2025)

## Abstract (arXiv)

We consider a model-based approach to perform batch off-policy evaluation in reinforcement learning. Our method takes a mixture-of-experts approach to combine parametric and non-parametric models of the environment such that the final value estimate has the least expected error. We do so by first estimating the local accuracy of each model and then using a planner to select which model to use at every time step as to minimize the return error estimate along entire trajectories. Across a variety of domains, our mixture-based approach outperforms the individual models alone as well as state-of-the-art importance sampling-based estimators.