# Online Recommendations for Agents with Discounted Adaptive Preferences

arXiv: https://arxiv.org/abs/2302.06014 · HTML: https://mlsanity.com/paper/2302.06014/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 12 Feb 2023.

## Builds on (most cited first, 18 of 18)

- [Introduction to Online Convex Optimization](https://mlsanity.com/paper/1909.05207/) (2019)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Feedback Loop and Bias Amplification in Recommender Systems](https://mlsanity.com/paper/2007.13019/) (2020)
- [Managing Popularity Bias in Recommender Systems with Personalized Re-ranking](https://mlsanity.com/paper/1901.07555/) (2019)
- [Rotting Bandits](https://mlsanity.com/paper/1702.07274/) (2017)
- [Preference Dynamics Under Personalized Recommendations](https://mlsanity.com/paper/2205.13026/) (2022)
- [Rebounding Bandits for Modeling Satiation Effects](https://mlsanity.com/paper/2011.06741/) (2020)
- [Polarization in Geometric Opinion Dynamics](https://mlsanity.com/paper/2106.12459/) (2021)
- [Towards Content Provider Aware Recommender Systems: A Simulation Study on the Interplay between User and Provider Utilities](https://mlsanity.com/paper/2105.02377/) (2021)
- [Diversified Recommendations for Agents with Adaptive Preferences](https://mlsanity.com/paper/2210.07773/) (2022)
- [A Last Switch Dependent Analysis of Satiation and Seasonality in Bandits](https://mlsanity.com/paper/2110.11819/) (2021)
- [Towards Psychologically-Grounded Dynamic Preference Models](https://mlsanity.com/paper/2208.01534/) (2022)
- [A Geometric Model of Opinion Polarization](https://mlsanity.com/paper/1910.05274/) (2019)
- [Bandit Learning with Positive Externalities](https://mlsanity.com/paper/1802.05693/) (2018)
- [Non-Stationary Bandits under Recharging Payoffs: Improved Planning with Sublinear Regret](https://mlsanity.com/paper/2205.14790/) (2022)
- [Congested Bandits: Optimal Routing via Short-term Resets](https://mlsanity.com/paper/2301.09251/) (2023)
- [Sequential Choice Bandits with Feedback for Personalizing users' experience](https://mlsanity.com/paper/2101.01572/) (2021)
- [Incentivized Bandit Learning with Self-Reinforcing User Preferences](https://mlsanity.com/paper/2105.08869/) (2021)

## Built on by (most cited first, 5 of 5)

- [Accounting for AI and Users Shaping One Another: The Role of Mathematical Models](https://mlsanity.com/paper/2404.12366/) (2024)
- [AI-Assisted Decision Making with Human Learning](https://mlsanity.com/paper/2502.13062/) (2025)
- [Impact of Decentralized Learning on Player Utilities in Stackelberg Games](https://mlsanity.com/paper/2403.00188/) (2024)
- [When Are Two Lists Better than One?: Benefits and Harms in Joint Decision-making](https://mlsanity.com/paper/2308.11721/) (2023)
- [User-Creator Feature Polarization in Recommender Systems with Dual Influence](https://mlsanity.com/paper/2407.14094/) (2024)

## Abstract (arXiv)

We consider a bandit recommendations problem in which an agent's preferences (representing selection probabilities over recommended items) evolve as a function of past selections, according to an unknown textitpreference model. In each round, we show a menu of k items (out of n total) to the agent, who then chooses a single item, and we aim to minimize regret with respect to some textittarget set (a subset of the item simplex) for adversarial losses over the agent's choices. Extending the setting from Agarwal and Brown (2022), where uniform-memory agents were considered, here we allow for non-uniform memory in which a discount factor is applied to the agent's memory vector at each subsequent round. In the "long-term memory" regime (when the effective memory horizon scales with T sublinearly), we show that efficient sublinear regret is obtainable with respect to the set of textiteverywhere instantaneously realizable distributions (the "EIRD set", as formulated in prior work) for any textitsmooth preference model. Further, for preferences which are bounded above and below by linear functions of memory weight (we call these "scale-bounded" preferences) we give an algorithm which obtains efficient sublinear regret with respect to nearly the textitentire item simplex. We show an NP-hardness result for expanding to targets beyond EIRD in general. In the "short-term memory" regime (when the memory horizon is constant), we show that scale-bounded preferences again enable efficient sublinear regret for nearly the entire simplex even without smoothness if losses do not change too frequently, yet we show an information-theoretic barrier for competing against the EIRD set under arbitrary smooth preference models even when losses are constant.