# Online Evaluation Methods for the Causal Effect of Recommendations

arXiv: https://arxiv.org/abs/2107.06630 · HTML: https://mlsanity.com/paper/2107.06630/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 35 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 9 of 9)

- [Recommendations as Treatments: Debiasing Learning and Evaluation](https://mlsanity.com/paper/1602.05352/) (2016)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [Causal Embeddings for Recommendation](https://mlsanity.com/paper/1706.07639/) (2017)
- [Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedback](https://mlsanity.com/paper/1909.03601/) (2019)
- [Measuring the Business Value of Recommender Systems](https://mlsanity.com/paper/1908.08328/) (2019)
- [Estimating the Causal Impact of Recommendation Systems from Observational Data](https://mlsanity.com/paper/1510.05569/) (2015)
- [Unbiased Learning for the Causal Effect of Recommendation](https://mlsanity.com/paper/2008.04563/) (2020)
- [Causality-Aware Neighborhood Methods for Recommender Systems](https://mlsanity.com/paper/2012.09442/) (2020)
- [Greedy Optimized Multileaving for Personalization](https://mlsanity.com/paper/1907.08346/) (2019)

## Built on by (most cited first, 1 of 1)

- [The Importance of Causality in Decision Making: A Perspective on Recommender Systems](https://mlsanity.com/paper/2410.01822/) (2024)

## Abstract (arXiv)

Evaluating the causal effect of recommendations is an important objective because the causal effect on user interactions can directly leads to an increase in sales and user engagement. To select an optimal recommendation model, it is common to conduct A/B testing to compare model performance. However, A/B testing of causal effects requires a large number of users, making such experiments costly and risky. We therefore propose the first interleaving methods that can efficiently compare recommendation models in terms of causal effects. In contrast to conventional interleaving methods, we measure the outcomes of both items on an interleaved list and items not on the interleaved list, since the causal effect is the difference between outcomes with and without recommendations. To ensure that the evaluations are unbiased, we either select items with equal probability or weight the outcomes using inverse propensity scores. We then verify the unbiasedness and efficiency of online evaluation methods through simulated online experiments. The results indicate that our proposed methods are unbiased and that they have superior efficiency to A/B testing.