# Advantage Amplification in Slowly Evolving Latent-State Environments

arXiv: https://arxiv.org/abs/1905.13559 · HTML: https://mlsanity.com/paper/1905.13559/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2019.

## Built on by (most cited first, 4 of 4)

- [Estimating and Penalizing Induced Preference Shifts in Recommender Systems](https://mlsanity.com/paper/2204.11966/) (2022)
- [RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems](https://mlsanity.com/paper/2103.08057/) (2021)
- [Reinforcement Learning with History-Dependent Dynamic Contexts](https://mlsanity.com/paper/2302.02061/) (2023)
- [Explicit User Manipulation in Reinforcement Learning Based Recommender Systems](https://mlsanity.com/paper/2203.10629/) (2022)

## Abstract (arXiv)

Latent-state environments with long horizons, such as those faced by recommender systems, pose significant challenges for reinforcement learning (RL). In this work, we identify and analyze several key hurdles for RL in such environments, including belief state error and small action advantage. We develop a general principle of advantage amplification that can overcome these hurdles through the use of temporal abstraction. We propose several aggregation methods and prove they induce amplification in certain settings. We also bound the loss in optimality incurred by our methods in environments where latent state evolves slowly and demonstrate their performance empirically in a stylized user-modeling task.