# Estimating and Penalizing Induced Preference Shifts in Recommender Systems

arXiv: https://arxiv.org/abs/2204.11966 · HTML: https://mlsanity.com/paper/2204.11966/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 58 references that are in the map.
- First posted on arXiv 25 Apr 2022.

## Builds on (most cited first, 25 of 25)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [Deep Learning based Recommender System: A Survey and New Perspectives](https://mlsanity.com/paper/1707.07435/) (2017)
- [How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility](https://mlsanity.com/paper/1710.11214/) (2017)
- [Feedback Loop and Bias Amplification in Recommender Systems](https://mlsanity.com/paper/2007.13019/) (2020)
- [Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning](https://mlsanity.com/paper/1805.10000/) (2018)
- [Degenerate Feedback Loops in Recommender Systems](https://mlsanity.com/paper/1902.10730/) (2019)
- [Generative Adversarial User Model for Reinforcement Learning Based Recommendation System](https://mlsanity.com/paper/1812.10613/) (2018)
- [Horizon: Facebook's Open Source Applied Reinforcement Learning Platform](https://mlsanity.com/paper/1811.00260/) (2018)
- [Hidden Incentives for Auto-Induced Distributional Shift](https://mlsanity.com/paper/2009.09153/) (2020)
- [Penalizing side effects using stepwise relative reachability](https://mlsanity.com/paper/1806.01186/) (2018)
- [Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation](https://mlsanity.com/paper/1911.03845/) (2019)
- [Reinforcement Learning for Slate-based Recommender Systems: A Tractable Decomposition and Practical Methodology](https://mlsanity.com/paper/1905.12767/) (2019)
- [Measuring Recommender System Effects with Simulated Users](https://mlsanity.com/paper/2101.04526/) (2021)
- [Digital Nudging with Recommender Systems: Survey and Future Directions](https://mlsanity.com/paper/2011.03413/) (2020)
- [Deep Reinforcement Learning with Attention for Slate Markov Decision Processes with High-Dimensional States and Actions](https://mlsanity.com/paper/1512.01124/) (2015)
- [Recognising the importance of preference change: A call for a coordinated multidisciplinary research effort in the age of AI](https://mlsanity.com/paper/2203.10525/) (2022)
- [Partially Observable Markov Decision Process for Recommender Systems](https://mlsanity.com/paper/1608.07793/) (2016)
- [Auditing E-Commerce Platforms for Algorithmically Curated Vaccine Misinformation](https://mlsanity.com/paper/2101.08419/) (2021)
- [Designing Recommender Systems to Depolarize](https://mlsanity.com/paper/2107.04953/) (2021)
- [REALab: An Embedded Perspective on Tampering](https://mlsanity.com/paper/2011.08820/) (2020)
- [Advantage Amplification in Slowly Evolving Latent-State Environments](https://mlsanity.com/paper/1905.13559/) (2019)
- [Unknowable Manipulators: Social Network Curator Algorithms](https://mlsanity.com/paper/1701.04895/) (2017)

## Built on by (most cited first, 29 of 29)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Harms from Increasingly Agentic Algorithmic Systems](https://mlsanity.com/paper/2302.10329/) (2023)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [Feedback Loops With Language Models Drive In-Context Reward Hacking](https://mlsanity.com/paper/2402.06627/) (2024)
- [A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)](https://mlsanity.com/paper/2404.00579/) (2024)
- [Beyond Preferences in AI Alignment](https://mlsanity.com/paper/2408.16984/) (2024)
- [On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback](https://mlsanity.com/paper/2411.02306/) (2024)
- [AI Alignment with Changing and Influenceable Reward Functions](https://mlsanity.com/paper/2405.17713/) (2024)
- [Discovering Agents](https://mlsanity.com/paper/2208.08345/) (2022)
- [MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking](https://mlsanity.com/paper/2501.13011/) (2025)
- [Improved Bayes Risk Can Yield Reduced Social Welfare Under Competition](https://mlsanity.com/paper/2306.14670/) (2023)
- [Performative Recommendation: Diversifying Content via Strategic Incentives](https://mlsanity.com/paper/2302.04336/) (2023)
- [Accounting for AI and Users Shaping One Another: The Role of Mathematical Models](https://mlsanity.com/paper/2404.12366/) (2024)
- [The Lock-in Hypothesis: Stagnation by Algorithm](https://mlsanity.com/paper/2506.06166/) (2025)
- [Implicit meta-learning may lead language models to trust more reliable sources](https://mlsanity.com/paper/2310.15047/) (2023)
- [Learning under Imitative Strategic Behavior with Unforeseeable Outcomes](https://mlsanity.com/paper/2405.01797/) (2024)
- [Incentivizing High-Quality Content in Online Recommender Systems](https://mlsanity.com/paper/2306.07479/) (2023)
- [Can Probabilistic Feedback Drive User Impacts in Online Platforms?](https://mlsanity.com/paper/2401.05304/) (2024)
- [System-2 Recommenders: Disentangling Utility and Engagement in Recommendation Systems via Temporal Point-Processes](https://mlsanity.com/paper/2406.01611/) (2024)
- [Lookahead Counterfactual Fairness](https://mlsanity.com/paper/2412.01065/) (2024)
- [Solutions to preference manipulation in recommender systems require knowledge of meta-preferences](https://mlsanity.com/paper/2209.11801/) (2022)
- [Optimising Human-AI Collaboration by Learning Convincing Explanations](https://mlsanity.com/paper/2311.07426/) (2023)
- [Inclusive Ethical Design for Recommender Systems](https://mlsanity.com/paper/2209.13021/) (2022)
- [Understanding or Manipulation: Rethinking Online Performance Gains of Modern Recommender Systems](https://mlsanity.com/paper/2210.05662/) (2022)
- [ARTAI: An Evaluation Platform to Assess Societal Risk of Recommender Algorithms](https://mlsanity.com/paper/2409.12396/) (2024)
- [Evolutionary Prediction Games](https://mlsanity.com/paper/2503.03401/) (2025)
- [Constructing Political Coordinates: Aggregating Over the Opposition for Diverse News Recommendation](https://mlsanity.com/paper/2511.17574/) (2025)
- [Compass: Continuously Aligning Social Media Feeds via In-Situ Reflections](https://mlsanity.com/paper/2608.04274/) (2026)
- [Deep Weighted Bellman Residual Minimization for Q^* Estimation](https://mlsanity.com/paper/2609.34593/) (2026)

## Abstract (arXiv)

The content that a recommender system (RS) shows to users influences them. Therefore, when choosing a recommender to deploy, one is implicitly also choosing to induce specific internal states in users. Even more, systems trained via long-horizon optimization will have direct incentives to manipulate users: in this work, we focus on the incentive to shift user preferences so they are easier to satisfy. We argue that - before deployment - system designers should: estimate the shifts a recommender would induce; evaluate whether such shifts would be undesirable; and perhaps even actively optimize to avoid problematic shifts. These steps involve two challenging ingredients: estimation requires anticipating how hypothetical algorithms would influence user preferences if deployed - we do this by using historical user interaction data to train a predictive user model which implicitly contains their preference dynamics; evaluation and optimization additionally require metrics to assess whether such influences are manipulative or otherwise unwanted - we use the notion of "safe shifts", that define a trust region within which behavior is safe: for instance, the natural way in which users would shift without interference from the system could be deemed "safe". In simulated experiments, we show that our learned preference dynamics model is effective in estimating user preferences and how they would respond to new recommenders. Additionally, we show that recommenders that optimize for staying in the trust region can avoid manipulative behaviors while still generating engagement.