# A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Markets

arXiv: https://arxiv.org/abs/2202.10574 · HTML: https://mlsanity.com/paper/2202.10574/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 100 references that are in the map.
- First posted on arXiv 21 Feb 2022.

## Builds on (most cited first, 30 of 41)

- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections](https://mlsanity.com/paper/1906.04733/) (2019)
- [A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation](https://mlsanity.com/paper/1806.02450/) (2018)
- [Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation](https://mlsanity.com/paper/1810.12429/) (2018)
- [A Theoretical Analysis of Deep Q-Learning](https://mlsanity.com/paper/1901.00137/) (2019)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [A Survey on Causal Inference](https://mlsanity.com/paper/2002.02770/) (2020)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [Minimax Weight and Q-Function Learning for Off-Policy Evaluation](https://mlsanity.com/paper/1910.12809/) (2019)
- [Finite-Sample Analysis for SARSA with Linear Function Approximation](https://mlsanity.com/paper/1902.02234/) (2019)
- [Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning](https://mlsanity.com/paper/1909.05850/) (2019)
- [Batch Policy Learning in Average Reward Markov Decision Processes](https://mlsanity.com/paper/2007.11771/) (2020)
- [Estimating Dynamic Treatment Regimes in Mobile Health Using V-learning](https://mlsanity.com/paper/1611.03531/) (2016)
- [Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settings](https://mlsanity.com/paper/2001.04515/) (2020)
- [Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation](https://mlsanity.com/paper/1910.07186/) (2019)
- [Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health](https://mlsanity.com/paper/1912.13088/) (2019)
- [Provably Efficient Causal Reinforcement Learning with Confounded Observational Data](https://mlsanity.com/paper/2006.12311/) (2020)
- [Deeply-Debiased Off-Policy Interval Estimation](https://mlsanity.com/paper/2105.04646/) (2021)
- [On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluation](https://mlsanity.com/paper/2201.06169/) (2022)
- [Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Making](https://mlsanity.com/paper/2002.01751/) (2020)
- [Program Evaluation and Causal Inference with High-Dimensional Data](https://mlsanity.com/paper/1311.2645/) (2013)
- [Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework](https://mlsanity.com/paper/2002.01711/) (2020)
- [Learning Optimal Distributionally Robust Individualized Treatment Rules](https://mlsanity.com/paper/2006.15121/) (2020)
- [Assessing Time-Varying Causal Effect Moderation in Mobile Health](https://mlsanity.com/paper/1601.00237/) (2016)
- [Causal Inference Under Interference And Network Uncertainty](https://mlsanity.com/paper/1907.00221/) (2019)
- [Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learning](https://mlsanity.com/paper/2110.10719/) (2021)
- [Time series experiments and causal estimands: exact randomization tests and trading](https://mlsanity.com/paper/1706.07840/) (2017)

## Built on by (most cited first, 4 of 4)

- [Policy Evaluation for Temporal and/or Spatial Dependent Experiments](https://mlsanity.com/paper/2202.10887/) (2022)
- [Evaluating Dynamic Conditional Quantile Treatment Effects with Applications in Ridesharing](https://mlsanity.com/paper/2305.10187/) (2023)
- [Two-sided Competing Matching Recommendation Markets With Quota and Complementary Preferences Constraints](https://mlsanity.com/paper/2301.10230/) (2023)
- [Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Data](https://mlsanity.com/paper/2403.11841/) (2024)

## Abstract (arXiv)

The two-sided markets such as ride-sharing companies often involve a group of subjects who are making sequential decisions across time and/or location. With the rapid development of smart phones and internet of things, they have substantially transformed the transportation landscape of human beings. In this paper we consider large-scale fleet management in ride-sharing companies that involve multiple units in different areas receiving sequences of products (or treatments) over time. Major technical challenges, such as policy evaluation, arise in those studies because (i) spatial and temporal proximities induce interference between locations and times; and (ii) the large number of locations results in the curse of dimensionality. To address both challenges simultaneously, we introduce a multi-agent reinforcement learning (MARL) framework for carrying policy evaluation in these studies. We propose novel estimators for mean outcomes under different products that are consistent despite the high-dimensionality of state-action space. The proposed estimator works favorably in simulation experiments. We further illustrate our method using a real dataset obtained from a two-sided marketplace company to evaluate the effects of applying different subsidizing policies. A Python implementation of our proposed method is available at https://github.com/RunzheStat/CausalMARL.