# Explicit User Manipulation in Reinforcement Learning Based Recommender Systems

arXiv: https://arxiv.org/abs/2203.10629 · HTML: https://mlsanity.com/paper/2203.10629/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 49 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Builds on (most cited first, 20 of 20)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning](https://mlsanity.com/paper/1910.11956/) (2019)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation](https://mlsanity.com/paper/1807.01473/) (2018)
- [RecSim: A Configurable Simulation Platform for Recommender Systems](https://mlsanity.com/paper/1909.04847/) (2019)
- [RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising](https://mlsanity.com/paper/1808.00720/) (2018)
- [Generative Adversarial User Model for Reinforcement Learning Based Recommendation System](https://mlsanity.com/paper/1812.10613/) (2018)
- [Partially Observable Markov Decision Processes (POMDPs) and Robotics](https://mlsanity.com/paper/2107.07599/) (2021)
- [Hidden Incentives for Auto-Induced Distributional Shift](https://mlsanity.com/paper/2009.09153/) (2020)
- [News Recommender System: A review of recent progress, challenges, and opportunities](https://mlsanity.com/paper/2009.04964/) (2020)
- [User Tampering in Reinforcement Learning Recommender Systems](https://mlsanity.com/paper/2109.04083/) (2021)
- [Optimal Recommendation to Users that React: Online Learning for a Class of POMDPs](https://mlsanity.com/paper/1603.09233/) (2016)
- [Modeling confirmation bias and polarization](https://mlsanity.com/paper/1607.00022/) (2016)
- [Partially Observable Markov Decision Process for Recommender Systems](https://mlsanity.com/paper/1608.07793/) (2016)
- [Designing Recommender Systems to Depolarize](https://mlsanity.com/paper/2107.04953/) (2021)
- [Advantage Amplification in Slowly Evolving Latent-State Environments](https://mlsanity.com/paper/1905.13559/) (2019)
- [A Model for the Influence of Media on the Ideology of Content in Online Social Networks](https://mlsanity.com/paper/1904.09238/) (2019)

## Abstract (arXiv)

Recommender systems are highly prevalent in the modern world due to their value to both users and platforms and services that employ them. Generally, they can improve the user experience and help to increase satisfaction, but they do not come without risks. One such risk is that of their effect on users and their ability to play an active role in shaping user preferences. This risk is more significant for reinforcement learning based recommender systems. These are capable of learning for instance, how recommended content shown to a user today may tamper that user's preference for other content recommended in the future. Reinforcement learning based recommendation systems can thus implicitly learn to influence users if that means maximizing clicks, engagement, or consumption. On social news and media platforms, in particular, this type of behavior is cause for alarm. Social media undoubtedly plays a role in public opinion and has been shown to be a contributing factor to increased political polarization. Recommender systems on such platforms, therefore, have great potential to influence users in undesirable ways. However, it may also be possible for this form of manipulation to be used intentionally. With advancements in political opinion dynamics modeling and larger collections of user data, explicit user manipulation in which the beliefs and opinions of users are tailored towards a certain end emerges as a significant concern in reinforcement learning based recommender systems.