mlsanityData to 7 Oct 2026

Markovian Interference in Experiments

arXiv 2206.02371 ↗First posted 6 Jun 2022cs.LG econ.EM stat.MLcited by 22

Not on the map. Why?

Abstract (arXiv)

We consider experiments in dynamical systems where interventions on some experimental units impact other units through a limiting constraint (such as a limited inventory). Despite outsize practical importance, the best estimators for this `Markovian' interference problem are largely heuristic in nature, and their bias is not well understood. We formalize the problem of inference in such experiments as one of policy evaluation. Off-policy estimators, while unbiased, apparently incur a large penalty in variance relative to state-of-the-art heuristics. We introduce an on-policy estimator: the Differences-In-Q's (DQ) estimator. We show that the DQ estimator can in general have exponentially smaller variance than off-policy evaluation. At the same time, its bias is second order in the impact of the intervention. This yields a striking bias-variance tradeoff so that the DQ estimator effectively dominates state-of-the-art alternatives. From a theoretical perspective, we introduce three separate novel techniques that are of independent interest in the theory of Reinforcement Learning (RL). Our empirical evaluation includes a set of experiments on a city-scale ride-hailing simulator.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 26 of its 76 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

26 of 76 references.

References · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  4. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  5. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  6. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  7. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  8. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  9. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  10. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  11. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  12. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
14 more
  1. 2013Graph cluster randomization: network exposure to multiple universescited by 42
  2. 2014Design and analysis of experiments in networks: Reducing bias from interferencecited by 41
  3. 2020Experimental Design in Two-Sided Platforms: An Analysis of Biascited by 39
  4. 2020Design and Analysis of Switchback Experimentscited by 37
  5. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
  6. 2020Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Frameworkcited by 29
  7. 2021Multiple Randomization Designs: Estimation and Inference with Interferencecited by 20
  8. 2017A framework for Multi-A(rmed)/B(andit) testing with online FDR controlcited by 17
  9. 2018Bipartite Causal Inference with Interferencecited by 15
  10. 2020Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approachcited by 15
  11. 2017Boosting the Actor with Dual Criticcited by 14
  12. 2015Model-assisted design of experiments in the presence of network correlated outcomescited by 9
  13. 2022Synthetically Controlled Banditscited by 8
  14. 2021Learning Treatment Effects in Panels with General Intervention Patternscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 5; 2025: 8; 2026: 4.

22 papers.

Sort:

By area:

  1. 2023Correcting for Interference in Experiments: A Case Study at Douyincited by 12
  2. 2024Multi-Armed Bandits with Interferencecited by 10
  3. 2023On the Statistical Benefits of Temporal Difference Learningcited by 8
  4. 2023Causal Message Passing for Experiments with Unknown and General Network Interferencecited by 7
  5. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  6. 2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
  7. 2024Optimal Adaptive Experimental Design for Estimating Treatment Effectcited by 5
  8. 2025Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approachcited by 5
  9. 2024Higher-Order Causal Message Passing for Experimentation with Complex Interferencecited by 4
  10. 2025Double Machine Learning for Causal Inference under Shared-State Interferencecited by 4
  11. 2025Choosing the Better Bandit Algorithm under Data Sharing: When Do A/B Experiments Work?cited by 4
  12. 2023Tackling Interference Induced by Data Training Loops in A/B Tests: A Weighted Training Approachcited by 3
10 more
  1. 2023Clustered Switchback Designs for Experimentation Under Spatio-temporal Interferencecited by 3
  2. 2025Simulating and Experimenting with Social Media Mobilization Using LLM Agentscited by 3
  3. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  4. 2025A Two-armed Bandit Framework for A/B Testingcited by 2
  5. 2025Differences-in-Neighbors for Network Interference in Experimentscited by 1
  6. 2025On Evolution-Based Models for Experimentation Under Interferencecited by 1
  7. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1
  8. 2026Design Experiments to Compare Multi-armed Bandit Algorithmscited by 1
  9. 2026Causal Effects with Unobserved Unit Types in Interacting Human-AI Systemscited by 0
  10. 2026Robust Sequential Experimental Design for A/B Testingcited by 0