mlsanityData to 7 Oct 2026

Combining Experimental and Historical Data for Policy Evaluation

arXiv 2406.00317 ↗First posted 1 Jun 2024stat.ML cs.LG stat.MEcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper studies policy evaluation with multiple data sources, especially in scenarios that involve one experimental dataset with two arms, complemented by a historical dataset generated under a single control arm. We propose novel data integration methods that linearly integrate base policy value estimators constructed based on the experimental and historical data, with weights optimized to minimize the mean square error (MSE) of the resulting combined estimator. We further apply the pessimistic principle to obtain more robust estimators, and extend these developments to sequential decision making. Theoretically, we establish non-asymptotic error bounds for the MSEs of our proposed estimators, and derive their oracle, efficiency and robustness properties across a broad spectrum of reward shift scenarios. Numerical experiments and real-data-based analyses from a ridesharing company demonstrate the superior performance of the proposed estimators.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 68 of its 122 references that are in the map. First posted on arXiv 1 Jun 2024.

Builds on

68 of 122 references.

References · 68

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  4. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  5. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  8. 2019Batch Policy Learning under Constraintscited by 242
  9. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  10. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  11. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  12. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
38 more
  1. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  2. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  3. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  4. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  5. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  6. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  7. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  8. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  9. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  10. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  11. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  12. 2018Removing Hidden Confounding by Experimental Groundingcited by 64
  13. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  14. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 63
  15. 2018Orthogonal Random Forest for Causal Inferencecited by 51
  16. 2020Causal inference methods for combining randomized trials and observational studies: a reviewcited by 50
  17. 2021A Deep Value-network Based Approach for Multi-Driver Order Dispatchingcited by 45
  18. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  19. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  20. 2020Design and Analysis of Switchback Experimentscited by 37
  21. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  22. 2021A Review of Generalizability and Transportabilitycited by 36
  23. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  24. 2021Reinforcement Learning for Ridesharing: An Extended Surveycited by 35
  25. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  26. 2021Post-Contextual-Bandit Inferencecited by 32
  27. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  28. 2018Combining multiple observational data sources to estimate causal effectscited by 29
  29. 2020Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Frameworkcited by 29
  30. 2017Nonparametric causal effects based on incremental propensity score interventionscited by 28
  31. 2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
  32. 2021Adaptive Combination of Randomized and Observational Datacited by 22
  33. 2022Markovian Interference in Experimentscited by 22
  34. 2017Time series experiments and causal estimands: exact randomization tests and tradingcited by 21
  35. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  36. 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
  37. 2021Federated Adaptive Causal Estimation (FACE) of Target Treatment Effectscited by 16
  38. 2023Multiply Robust Federated Estimation of Targeted Average Treatment Effectscited by 13

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  2. 2025A Two-armed Bandit Framework for A/B Testingcited by 2
  3. 2025Multi-task Learning for Heterogeneous Multi-source Block-Wise Missing Datacited by 1
  4. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1
  5. 2025Multi-Task Learning with Covariate-Overlap Regularizationcited by 0
  6. 2026Robust Sequential Experimental Design for A/B Testingcited by 0