# Examining Policy Entropy of Reinforcement Learning Agents for Personalization Tasks

arXiv: https://arxiv.org/abs/2211.11869 · HTML: https://mlsanity.com/paper/2211.11869/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 58 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Bridging the Gap Between Value and Policy Based Reinforcement Learning](https://mlsanity.com/paper/1702.08892/) (2017)
- [Understanding the impact of entropy on policy optimization](https://mlsanity.com/paper/1811.11214/) (2018)
- [Neural Policy Gradient Methods: Global Optimality and Rates of Convergence](https://mlsanity.com/paper/1909.01150/) (2019)
- [RecSim: A Configurable Simulation Platform for Recommender Systems](https://mlsanity.com/paper/1909.04847/) (2019)
- [RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising](https://mlsanity.com/paper/1808.00720/) (2018)
- [Exploration in Interactive Personalized Music Recommendation: A Reinforcement Learning Approach](https://mlsanity.com/paper/1311.6355/) (2013)
- [Policy Gradients for Contextual Recommendations](https://mlsanity.com/paper/1802.04162/) (2018)
- [The Alberta Plan for AI Research](https://mlsanity.com/paper/2208.11173/) (2022)
- [An Actor-Critic Contextual Bandit Algorithm for Personalized Mobile Health Interventions](https://mlsanity.com/paper/1706.09090/) (2017)
- [Optimization Issues in KL-Constrained Approximate Policy Iteration](https://mlsanity.com/paper/2102.06234/) (2021)
- [Robust Actor-Critic Contextual Bandit for Mobile Health (mHealth) Interventions](https://mlsanity.com/paper/1802.09714/) (2018)
- [An Alternate Policy Gradient Estimator for Softmax Policies](https://mlsanity.com/paper/2112.11622/) (2021)
- [A clustering-based reinforcement learning approach for tailored personalization of e-Health interventions](https://mlsanity.com/paper/1804.03592/) (2018)
- [Supervised Advantage Actor-Critic for Recommender Systems](https://mlsanity.com/paper/2111.03474/) (2021)
- [A Notation for Markov Decision Processes](https://mlsanity.com/paper/1512.09075/) (2015)
- [Offline Policy Comparison under Limited Historical Agent-Environment Interactions](https://mlsanity.com/paper/2106.03934/) (2021)
- [On the Unreasonable Efficiency of State Space Clustering in Personalization Tasks](https://mlsanity.com/paper/2112.13141/) (2021)

## Built on by (most cited first, 3 of 3)

- [Mamba for Scalable and Efficient Personalized Recommendations](https://mlsanity.com/paper/2409.17165/) (2024)
- [Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards](https://mlsanity.com/paper/2410.17126/) (2024)
- [Increasing Entropy to Boost Policy Gradient Performance on Personalization Tasks](https://mlsanity.com/paper/2310.05324/) (2023)

## Abstract (arXiv)

This effort is focused on examining the behavior of reinforcement learning systems in personalization environments and detailing the differences in policy entropy associated with the type of learning algorithm utilized. We demonstrate that Policy Optimization agents often possess low-entropy policies during training, which in practice results in agents prioritizing certain actions and avoiding others. Conversely, we also show that Q-Learning agents are far less susceptible to such behavior and generally maintain high-entropy policies throughout training, which is often preferable in real-world applications. We provide a wide range of numerical experiments as well as theoretical justification to show that these differences in entropy are due to the type of learning being employed.