mlsanityData to 13 Oct 2026

A Reinforcement Learning Approach to Estimating Long-term Treatment Effects

arXiv 2210.07536 ↗First posted 14 Oct 2022cs.LG stat.ME stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Randomized experiments (a.k.a. A/B tests) are a powerful tool for estimating treatment effects, to inform decisions making in business, healthcare and other applications. In many problems, the treatment has a lasting effect that evolves over time. A limitation with randomized experiments is that they do not easily extend to measure long-term effects, since running long experiments is time-consuming and expensive. In this paper, we take a reinforcement learning (RL) approach that estimates the average reward in a Markov process. Motivated by real-world scenarios where the observed state transition is nonstationary, we develop a new algorithm for a class of nonstationary problems, and demonstrate promising results in two synthetic datasets and one online store dataset.

Key facts. Cited by 5 papers in the map (data through 13 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

20 of 36 references.

References · 20

  1. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  2. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  3. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  4. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  5. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  6. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  7. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  8. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 107
  9. 2019A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Normcited by 105
  10. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
  11. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  12. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
8 more
  1. 2021Universal Off-Policy Evaluationcited by 43
  2. 2020Deep Reinforcement Learning amidst Lifelong Non-Stationaritycited by 42
  3. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 40
  4. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 35
  5. 2020Using Experiments to Correct for Selection in Observational Studiescited by 28
  6. 2021Provable RL with Exogenous Distractors via Multistep Inverse Dynamicscited by 28
  7. 2018Discovering and Removing Exogenous State Variables and Rewards for Reinforcement Learningcited by 24
  8. 2019Learning Compact Models for Planning with Exogenous Processescited by 11

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 4; 2025: 0; 2026: 1.

5 papers.

  1. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  2. 2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
  3. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1
  4. 2024Short-Long Policy Evaluation with Novel Actionscited by 0
  5. 2024Predicting Long Term Sequential Policy Value Using Softer Surrogatescited by 0