# Measuring Recommender System Effects with Simulated Users arXiv: https://arxiv.org/abs/2101.04526 ยท HTML: https://mlsanity.com/paper/2101.04526/ ## Key facts - Cited by 20 papers in the map (data through 14 Oct 2026). - Builds on 16 of its 66 references that are in the map. - First posted on arXiv 12 Jan 2021. ## Builds on (most cited first, 16 of 16) - [Fairness of Exposure in Rankings](https://mlsanity.com/paper/1802.07281/) (2018) - [Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches](https://mlsanity.com/paper/1907.06902/) (2019) - [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018) - [Fairness in Recommendation Ranking through Pairwise Comparisons](https://mlsanity.com/paper/1903.00780/) (2019) - [Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks](https://mlsanity.com/paper/1708.04617/) (2017) - [Equity of Attention: Amortizing Individual Fairness in Rankings](https://mlsanity.com/paper/1805.01788/) (2018) - [How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility](https://mlsanity.com/paper/1710.11214/) (2017) - [RecSim: A Configurable Simulation Platform for Recommender Systems](https://mlsanity.com/paper/1909.04847/) (2019) - [Managing Popularity Bias in Recommender Systems with Personalized Re-ranking](https://mlsanity.com/paper/1901.07555/) (2019) - [Cascading Bandits: Learning to Rank in the Cascade Model](https://mlsanity.com/paper/1502.02763/) (2015) - [Degenerate Feedback Loops in Recommender Systems](https://mlsanity.com/paper/1902.10730/) (2019) - [Auditing Radicalization Pathways on YouTube](https://mlsanity.com/paper/1908.08313/) (2019) - [Fighting Fire with Fire: Using Antidote Data to Improve Polarization and Fairness of Recommender Systems](https://mlsanity.com/paper/1812.01504/) (2018) - [Deep Reinforcement Learning for List-wise Recommendations](https://mlsanity.com/paper/1801.00209/) (2017) - [Recommendations and User Agency: The Reachability of Collaboratively-Filtered Information](https://mlsanity.com/paper/1912.10068/) (2019) - [Algorithmic Extremism: Examining YouTube's Rabbit Hole of Radicalization](https://mlsanity.com/paper/1912.11211/) (2019) ## Built on by (most cited first, 20 of 20) - [User Behavior Simulation with Large Language Model based Agents](https://mlsanity.com/paper/2306.02552/) (2023) - [Estimating and Penalizing Induced Preference Shifts in Recommender Systems](https://mlsanity.com/paper/2204.11966/) (2022) - [Preference Dynamics Under Personalized Recommendations](https://mlsanity.com/paper/2205.13026/) (2022) - [RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems](https://mlsanity.com/paper/2103.08057/) (2021) - [Fair ranking: a critical review, challenges, and future directions](https://mlsanity.com/paper/2201.12662/) (2022) - [Relevance meets Diversity: A User-Centric Framework for Knowledge Exploration through Recommendations](https://mlsanity.com/paper/2408.03772/) (2024) - [Quantifying Availability and Discovery in Recommender Systems via Stochastic Reachability](https://mlsanity.com/paper/2107.00833/) (2021) - [Learning from Negative User Feedback and Measuring Responsiveness for Sequential Recommenders](https://mlsanity.com/paper/2308.12256/) (2023) - [Synthetic Data-Based Simulators for Recommender Systems: A Survey](https://mlsanity.com/paper/2206.11338/) (2022) - [Algorithmic Drift: A Simulation Framework to Study the Effects of Recommender Systems on User Preferences](https://mlsanity.com/paper/2409.16478/) (2024) - [Manifestations of Xenophobia in AI Systems](https://mlsanity.com/paper/2212.07877/) (2022) - [Recency Dropout for Recurrent Recommender Systems](https://mlsanity.com/paper/2201.11016/) (2022) - [Minimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policies](https://mlsanity.com/paper/2409.17436/) (2024) - [Efficient and Responsible Adaptation of Large Language Models for Robust and Equitable Top-k Recommendations](https://mlsanity.com/paper/2501.04762/) (2025) - [Simulating News Recommendation Ecosystem for Fun and Profit](https://mlsanity.com/paper/2305.14103/) (2023) - [A Unified Causal Framework for Auditing Recommender Systems for Ethical Concerns](https://mlsanity.com/paper/2409.13210/) (2024) - [Neural Click Models for Recommender Systems](https://mlsanity.com/paper/2409.20055/) (2024) - [Understanding or Manipulation: Rethinking Online Performance Gains of Modern Recommender Systems](https://mlsanity.com/paper/2210.05662/) (2022) - [Modeling Social Media Recommendation Impacts Using Academic Networks: A Graph Neural Network Approach](https://mlsanity.com/paper/2410.04552/) (2024) - [The Diversity Paradox revisited: Systemic Effects of Feedback Loops in Recommender Systems](https://mlsanity.com/paper/2602.16315/) (2026) ## Abstract (arXiv) Imagine a food recommender system -- how would we check if it is causing and fostering unhealthy eating habits or merely reflecting users' interests? How much of a user's experience over time with a recommender is caused by the recommender system's choices and biases, and how much is based on the user's preferences and biases? Popularity bias and filter bubbles are two of the most well-studied recommender system biases, but most of the prior research has focused on understanding the system behavior in a single recommendation step. How do these biases interplay with user behavior, and what types of user experiences are created from repeated interactions? In this work, we offer a simulation framework for measuring the impact of a recommender system under different types of user behavior. Using this simulation framework, we can (a) isolate the effect of the recommender system from the user preferences, and (b) examine how the system performs not just on average for an "average user" but also the extreme experiences under atypical user behavior. As part of the simulation framework, we propose a set of evaluation metrics over the simulations to understand the recommender system's behavior. Finally, we present two empirical case studies -- one on traditional collaborative filtering in MovieLens and one on a large-scale production recommender system -- to understand how popularity bias manifests over time.