# Policy Learning with Adaptively Collected Data arXiv: https://arxiv.org/abs/2105.02344 ยท HTML: https://mlsanity.com/paper/2105.02344/ ## Key facts - Cited by 21 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 86 references that are in the map. - First posted on arXiv 5 May 2021. ## Builds on (most cited first, 22 of 22) - [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020) - [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014) - [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017) - [Is Pessimism Provably Efficient for Offline RL?](https://mlsanity.com/paper/2012.15085/) (2020) - [Learning to Optimize Via Posterior Sampling](https://mlsanity.com/paper/1301.2609/) (2013) - [Recommendations as Treatments: Debiasing Learning and Evaluation](https://mlsanity.com/paper/1602.05352/) (2016) - [Optimal and Adaptive Off-policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/1612.01205/) (2016) - [Off-policy evaluation for slate recommendation](https://mlsanity.com/paper/1605.04812/) (2016) - [Confidence Intervals for Policy Evaluation in Adaptive Experiments](https://mlsanity.com/paper/1911.02768/) (2019) - [Doubly robust off-policy evaluation with shrinkage](https://mlsanity.com/paper/1907.09623/) (2019) - [Confounding-Robust Policy Improvement](https://mlsanity.com/paper/1805.08593/) (2018) - [Offline Multi-Action Policy Learning: Generalization and Optimization](https://mlsanity.com/paper/1810.04778/) (2018) - [Off-policy Bandits with Deficient Support](https://mlsanity.com/paper/2006.09438/) (2020) - [Balanced Policy Evaluation and Learning](https://mlsanity.com/paper/1705.07384/) (2017) - [Residual Weighted Learning for Estimating Individualized Treatment Rules](https://mlsanity.com/paper/1508.03179/) (2015) - [Why Adaptively Collected Data Have Negative Bias and How to Correct for It](https://mlsanity.com/paper/1708.01977/) (2017) - [Semi-Parametric Efficient Policy Learning with Continuous Actions](https://mlsanity.com/paper/1905.10116/) (2019) - [Predicting with Proxies: Transfer Learning in High Dimension](https://mlsanity.com/paper/1812.11097/) (2018) - [Are sample means in multi-armed bandits positively or negatively biased?](https://mlsanity.com/paper/1905.11397/) (2019) - [Estimation Considerations in Contextual Bandits](https://mlsanity.com/paper/1711.07077/) (2017) - [Dynamic Assortment Personalization in High Dimensions](https://mlsanity.com/paper/1610.05604/) (2016) - [Efficient Policy Learning from Surrogate-Loss Classification Reductions](https://mlsanity.com/paper/2002.05153/) (2020) ## Built on by (most cited first, 21 of 21) - [Doubly Robust Distributionally Robust Off-Policy Evaluation and Learning](https://mlsanity.com/paper/2202.09667/) (2022) - [Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequality](https://mlsanity.com/paper/2212.09900/) (2022) - [On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/2211.13208/) (2022) - [Non-Stationary Representation Learning in Sequential Linear Bandits](https://mlsanity.com/paper/2201.04805/) (2022) - [Off-policy estimation of linear functionals: Non-asymptotic theory for semi-parametric efficiency](https://mlsanity.com/paper/2209.13075/) (2022) - [Contextual Bandits in a Survey Experiment on Charitable Giving: Within-Experiment Outcomes versus Policy Learning](https://mlsanity.com/paper/2211.12004/) (2022) - [Learning Complementary Policies for Human-AI Teams](https://mlsanity.com/paper/2302.02944/) (2023) - [Sequential Counterfactual Risk Minimization](https://mlsanity.com/paper/2302.12120/) (2023) - [Adaptive Linear Estimating Equations](https://mlsanity.com/paper/2307.07320/) (2023) - [On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond](https://mlsanity.com/paper/2401.03301/) (2024) - [Distributionally Robust Policy Learning under Concept Drifts](https://mlsanity.com/paper/2412.14297/) (2024) - [A Statistical Framework for Alignment with Biased AI Feedback](https://mlsanity.com/paper/2602.08259/) (2026) - [Robust Offline Policy Learning with Observational Data from Multiple Sources](https://mlsanity.com/paper/2410.08537/) (2024) - [Off-policy estimation with adaptively collected data: the power of online learning](https://mlsanity.com/paper/2411.12786/) (2024) - [Adaptive Experimental Design for Policy Learning](https://mlsanity.com/paper/2401.03756/) (2024) - [Design Experiments to Compare Multi-armed Bandit Algorithms](https://mlsanity.com/paper/2603.05919/) (2026) - [Federated Offline Policy Learning](https://mlsanity.com/paper/2305.12407/) (2023) - [Fast Best-in-Class Regret for Contextual Bandits](https://mlsanity.com/paper/2510.15483/) (2025) - [Pessimistic Risk-Aware Policy Learning in Contextual Bandits](https://mlsanity.com/paper/2605.15620/) (2026) - [Wasserstein Policy Learning for Distributional Outcomes](https://mlsanity.com/paper/2606.19117/) (2026) - [A Hierarchy of Policy Learning Problems](https://mlsanity.com/paper/2607.03385/) (2026) ## Abstract (arXiv) Learning optimal policies from historical data enables personalization in a wide variety of applications including healthcare, digital recommendations, and online education. The growing policy learning literature focuses on settings where the data collection rule stays fixed throughout the experiment. However, adaptive data collection is becoming more common in practice, from two primary sources: 1) data collected from adaptive experiments that are designed to improve inferential efficiency; 2) data collected from production systems that progressively evolve an operational policy to improve performance over time (e.g. contextual bandits). Yet adaptivity complicates the optimal policy identification ex post, since samples are dependent, and each treatment may not receive enough observations for each type of individual. In this paper, we make initial research inquiries into addressing the challenges of learning the optimal policy with adaptively collected data. We propose an algorithm based on generalized augmented inverse propensity weighted (AIPW) estimators, which non-uniformly reweight the elements of a standard AIPW estimator to control worst-case estimation variance. We establish a finite-sample regret upper bound for our algorithm and complement it with a regret lower bound that quantifies the fundamental difficulty of policy learning with adaptive data. When equipped with the best weighting scheme, our algorithm achieves minimax rate optimal regret guarantees even with diminishing exploration. Finally, we demonstrate our algorithm's effectiveness using both synthetic data and public benchmark datasets.