mlsanityData to 7 Oct 2026

A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Markets

arXiv 2202.10574 ↗First posted 21 Feb 2022stat.ML cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

The two-sided markets such as ride-sharing companies often involve a group of subjects who are making sequential decisions across time and/or location. With the rapid development of smart phones and internet of things, they have substantially transformed the transportation landscape of human beings. In this paper we consider large-scale fleet management in ride-sharing companies that involve multiple units in different areas receiving sequences of products (or treatments) over time. Major technical challenges, such as policy evaluation, arise in those studies because (i) spatial and temporal proximities induce interference between locations and times; and (ii) the large number of locations results in the curse of dimensionality. To address both challenges simultaneously, we introduce a multi-agent reinforcement learning (MARL) framework for carrying policy evaluation in these studies. We propose novel estimators for mean outcomes under different products that are consistent despite the high-dimensionality of state-action space. The proposed estimator works favorably in simulation experiments. We further illustrate our method using a real dataset obtained from a two-sided marketplace company to evaluate the effects of applying different subsidizing policies. A Python implementation of our proposed method is available at https://github.com/RunzheStat/CausalMARL.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 41 of its 100 references that are in the map. First posted on arXiv 21 Feb 2022.

Builds on

41 of 100 references.

References · 41

  1. 2015Estimation and Inference of Heterogeneous Treatment Effects using Random Forestscited by 597
  2. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  3. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  4. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  5. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  6. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  7. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  8. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  9. 2018Mean Field Multi-Agent Reinforcement Learningcited by 202
  10. 2020A Survey on Causal Inferencecited by 202
  11. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  12. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
29 more
  1. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  2. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  3. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  4. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  5. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  6. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  7. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  8. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  9. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  10. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  11. 2020Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Makingcited by 30
  12. 2013Program Evaluation and Causal Inference with High-Dimensional Datacited by 29
  13. 2020Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Frameworkcited by 29
  14. 2020Learning Optimal Distributionally Robust Individualized Treatment Rulescited by 28
  15. 2016Assessing Time-Varying Causal Effect Moderation in Mobile Healthcited by 27
  16. 2019Causal Inference Under Interference And Network Uncertaintycited by 27
  17. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  18. 2017Time series experiments and causal estimands: exact randomization tests and tradingcited by 21
  19. 2016Interpretable Dynamic Treatment Regimescited by 17
  20. 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
  21. 2020Deep Jump Learning for Off-Policy Evaluation in Continuous Treatment Settingscited by 14
  22. 2016Taxi Dispatch with Real-Time Sensing Data in Metropolitan Areas: A Receding Horizon Control Approachcited by 13
  23. 2020A review of spatial causal inference methods for environmental and epidemiological applicationscited by 12
  24. 2015Using Decision Lists to Construct Interpretable and Parsimonious Treatment Regimescited by 11
  25. 2020Personalized Policy Learning using Longitudinal Mobile Health Datacited by 10
  26. 2014Constructing Dynamic Treatment Regimes in Infinite-Horizon Settingscited by 4
  27. 2018Bayesian method for causal inference in spatially-correlated multivariate time seriescited by 3
  28. 2019Resampling-based Confidence Intervals for Model-free Robust Inference on Optimal Treatment Regimescited by 3
  29. 2020Near-optimal Individualized Treatment Recommendationscited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 1.

4 papers.

  1. 2022Policy Evaluation for Temporal and/or Spatial Dependent Experimentscited by 8
  2. 2023Evaluating Dynamic Conditional Quantile Treatment Effects with Applications in Ridesharingcited by 7
  3. 2023Two-sided Competing Matching Recommendation Markets With Quota and Complementary Preferences Constraintscited by 5
  4. 2024Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Datacited by 0