# User Tampering in Reinforcement Learning Recommender Systems

arXiv: https://arxiv.org/abs/2109.04083 · HTML: https://mlsanity.com/paper/2109.04083/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 40 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 13 of 13)

- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [Deep Reinforcement Learning for Page-wise Recommendations](https://mlsanity.com/paper/1805.02343/) (2018)
- [FairRec: Two-Sided Fairness for Personalized Recommendations in Two-Sided Platforms](https://mlsanity.com/paper/2002.10764/) (2020)
- [Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning](https://mlsanity.com/paper/1802.06501/) (2018)
- [Degenerate Feedback Loops in Recommender Systems](https://mlsanity.com/paper/1902.10730/) (2019)
- [Agent Incentives: A Causal Perspective](https://mlsanity.com/paper/2102.01685/) (2021)
- [Horizon: Facebook's Open Source Applied Reinforcement Learning Platform](https://mlsanity.com/paper/1811.00260/) (2018)
- [Hidden Incentives for Auto-Induced Distributional Shift](https://mlsanity.com/paper/2009.09153/) (2020)
- [Deep Reinforcement Learning based Recommendation with Explicit User-Item Interactions Modeling](https://mlsanity.com/paper/1810.12027/) (2018)
- [What are you optimizing for? Aligning Recommender Systems with Human Values](https://mlsanity.com/paper/2107.10939/) (2021)
- [Reinforcement Learning-based Product Delivery Frequency Control](https://mlsanity.com/paper/2012.10858/) (2020)
- [Aligning Daily Activities with Personality: Towards A Recommender System for Improving Wellbeing](https://mlsanity.com/paper/1909.03847/) (2019)

## Built on by (most cited first, 23 of 23)

- [Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models](https://mlsanity.com/paper/2406.10162/) (2024)
- [Harms from Increasingly Agentic Algorithmic Systems](https://mlsanity.com/paper/2302.10329/) (2023)
- [On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback](https://mlsanity.com/paper/2411.02306/) (2024)
- [Discovering Agents](https://mlsanity.com/paper/2208.08345/) (2022)
- [Fairness in Recommendation: Foundations, Methods and Applications](https://mlsanity.com/paper/2205.13619/) (2022)
- [Preference Dynamics Under Personalized Recommendations](https://mlsanity.com/paper/2205.13026/) (2022)
- [The Reasons that Agents Act: Intention and Instrumental Goals](https://mlsanity.com/paper/2402.07221/) (2024)
- [Two Types of AI Existential Risk: Decisive and Accumulative](https://mlsanity.com/paper/2401.07836/) (2024)
- [MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking](https://mlsanity.com/paper/2501.13011/) (2025)
- [Reward Reports for Reinforcement Learning](https://mlsanity.com/paper/2204.10817/) (2022)
- [Recognising the importance of preference change: A call for a coordinated multidisciplinary research effort in the age of AI](https://mlsanity.com/paper/2203.10525/) (2022)
- [Accounting for AI and Users Shaping One Another: The Role of Mathematical Models](https://mlsanity.com/paper/2404.12366/) (2024)
- [A Complete Criterion for Value of Information in Soluble Influence Diagrams](https://mlsanity.com/paper/2202.11629/) (2022)
- [Long-Term Fairness Inquiries and Pursuits in Machine Learning: A Survey of Notions, Methods, and Challenges](https://mlsanity.com/paper/2406.06736/) (2024)
- [You Don't Bring Me Flowers: Mitigating Unwanted Recommendations Through Conformal Risk Control](https://mlsanity.com/paper/2507.16829/) (2025)
- [Appropriateness is all you need!](https://mlsanity.com/paper/2304.14553/) (2023)
- [Autonomy by Design: Preserving Human Autonomy in AI Decision-Support](https://mlsanity.com/paper/2506.23952/) (2025)
- [Solutions to preference manipulation in recommender systems require knowledge of meta-preferences](https://mlsanity.com/paper/2209.11801/) (2022)
- [The Alignment Bottleneck](https://mlsanity.com/paper/2509.15932/) (2025)
- [Explicit User Manipulation in Reinforcement Learning Based Recommender Systems](https://mlsanity.com/paper/2203.10629/) (2022)
- [Inclusive Ethical Design for Recommender Systems](https://mlsanity.com/paper/2209.13021/) (2022)
- [Evolutionary Prediction Games](https://mlsanity.com/paper/2503.03401/) (2025)
- [MultiScale Contextual Bandits for Long Term Objectives](https://mlsanity.com/paper/2503.17674/) (2025)

## Abstract (arXiv)

In this paper, we introduce new formal methods and provide empirical evidence to highlight a unique safety concern prevalent in reinforcement learning (RL)-based recommendation algorithms -- 'user tampering.' User tampering is a situation where an RL-based recommender system may manipulate a media user's opinions through its suggestions as part of a policy to maximize long-term user engagement. We use formal techniques from causal modeling to critically analyze prevailing solutions proposed in the literature for implementing scalable RL-based recommendation systems, and we observe that these methods do not adequately prevent user tampering. Moreover, we evaluate existing mitigation strategies for reward tampering issues, and show that these methods are insufficient in addressing the distinct phenomenon of user tampering within the context of recommendations. We further reinforce our findings with a simulation study of an RL-based recommendation system focused on the dissemination of political content. Our study shows that a Q-learning algorithm consistently learns to exploit its opportunities to polarize simulated users with its early recommendations in order to have more consistent success with subsequent recommendations that align with this induced polarization. Our findings emphasize the necessity for developing safer RL-based recommendation systems and suggest that achieving such safety would require a fundamental shift in the design away from the approaches we have seen in the recent literature.