# Estimating the Causal Impact of Recommendation Systems from Observational Data

arXiv: https://arxiv.org/abs/1510.05569 · HTML: https://mlsanity.com/paper/1510.05569/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2015.

## Built on by (most cited first, 21 of 21)

- [Fair ranking: a critical review, challenges, and future directions](https://mlsanity.com/paper/2201.12662/) (2022)
- [Causal Deep Learning](https://mlsanity.com/paper/2303.02186/) (2023)
- [Breaking Feedback Loops in Recommender Systems with Causal Inference](https://mlsanity.com/paper/2207.01616/) (2022)
- [A Review of the Role of Causality in Developing Trustworthy AI Systems](https://mlsanity.com/paper/2302.06975/) (2023)
- [Instrumental Variables in Causal Inference and Machine Learning: A Survey](https://mlsanity.com/paper/2212.05778/) (2022)
- [No Thoughts Just AI: Biased LLM Hiring Recommendations Alter Human Decision Making and Limit Human Autonomy](https://mlsanity.com/paper/2509.04404/) (2025)
- [Correcting the User Feedback-Loop Bias for Recommendation Systems](https://mlsanity.com/paper/2109.06037/) (2021)
- [In the Eye of the Beholder: Robust Prediction with Causal User Modeling](https://mlsanity.com/paper/2206.00416/) (2022)
- [Revisiting Reciprocal Recommender Systems: Metrics, Formulation, and Method](https://mlsanity.com/paper/2408.09748/) (2024)
- [Recommendation System Simulations: A Discussion of Two Key Challenges](https://mlsanity.com/paper/2109.02475/) (2021)
- [Score-matching-based Structure Learning for Temporal Data on Networks](https://mlsanity.com/paper/2412.07469/) (2024)
- [Online Evaluation Methods for the Causal Effect of Recommendations](https://mlsanity.com/paper/2107.06630/) (2021)
- [Identifying Causal Influences on Publication Trends and Behavior: A Case Study of the Computational Linguistics Community](https://mlsanity.com/paper/2110.07938/) (2021)
- [REST: Debiased Social Recommendation via Reconstructing Exposure Strategies](https://mlsanity.com/paper/2201.04952/) (2022)
- [A Unified Causal Framework for Auditing Recommender Systems for Ethical Concerns](https://mlsanity.com/paper/2409.13210/) (2024)
- [The Importance of Causality in Decision Making: A Perspective on Recommender Systems](https://mlsanity.com/paper/2410.01822/) (2024)
- [The RAIL Principles for Neurosymbolic AI: Reasoning, Assurances, Interfacing and Learning](https://mlsanity.com/paper/2608.04285/) (2026)
- [Evaluating Ensemble Methods for News Recommender Systems](https://mlsanity.com/paper/2406.16106/) (2024)
- [Introducing CausalBench: A Flexible Benchmark Framework for Causal Analysis and Machine Learning](https://mlsanity.com/paper/2409.08419/) (2024)
- [Counterfactual Forecasting of Human Behavior using Generative AI and Causal Graphs](https://mlsanity.com/paper/2511.07484/) (2025)
- [Reason-Mediated Behavioral Models for Auditing LLM Social Simulators](https://mlsanity.com/paper/2607.24649/) (2026)

## Abstract (arXiv)

Recommendation systems are an increasingly prominent part of the web, accounting for up to a third of all traffic on several of the world's most popular sites. Nevertheless, little is known about how much activity such systems actually cause over and above activity that would have occurred via other means (e.g., search) if recommendations were absent. Although the ideal way to estimate the causal impact of recommendations is via randomized experiments, such experiments are costly and may inconvenience users. In this paper, therefore, we present a method for estimating causal effects from purely observational data. Specifically, we show that causal identification through an instrumental variable is possible when a product experiences an instantaneous shock in direct traffic and the products recommended next to it do not. We then apply our method to browsing logs containing anonymized activity for 2.1 million users on Amazon.com over a 9 month period and analyze over 4,000 unique products that experience such shocks. We find that although recommendation click-throughs do account for a large fraction of traffic among these products, at least 75% of this activity would likely occur in the absence of recommendations. We conclude with a discussion about the assumptions under which the method is appropriate and caveats around extrapolating results to other products, sites, or settings.