# Increasing Entropy to Boost Policy Gradient Performance on Personalization Tasks

arXiv: https://arxiv.org/abs/2310.05324 · HTML: https://mlsanity.com/paper/2310.05324/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 51 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [MMD GAN: Towards Deeper Understanding of Moment Matching Network](https://mlsanity.com/paper/1705.08584/) (2017)
- [Understanding the impact of entropy on policy optimization](https://mlsanity.com/paper/1811.11214/) (2018)
- [Diversity-Driven Exploration Strategy for Deep Reinforcement Learning](https://mlsanity.com/paper/1802.04564/) (2018)
- [Information asymmetry in KL-regularized RL](https://mlsanity.com/paper/1905.01240/) (2019)
- [Diversity-Inducing Policy Gradient: Using Maximum Mean Discrepancy to Find a Set of Diverse Policies](https://mlsanity.com/paper/1906.00088/) (2019)
- [Counterfactual Risk Minimization: Learning from Logged Bandit Feedback](https://mlsanity.com/paper/1502.02362/) (2015)
- [Exploration in Interactive Personalized Music Recommendation: A Reinforcement Learning Approach](https://mlsanity.com/paper/1311.6355/) (2013)
- [An Actor-Critic Contextual Bandit Algorithm for Personalized Mobile Health Interventions](https://mlsanity.com/paper/1706.09090/) (2017)
- [Robust Actor-Critic Contextual Bandit for Mobile Health (mHealth) Interventions](https://mlsanity.com/paper/1802.09714/) (2018)
- [A clustering-based reinforcement learning approach for tailored personalization of e-Health interventions](https://mlsanity.com/paper/1804.03592/) (2018)
- [Examining Policy Entropy of Reinforcement Learning Agents for Personalization Tasks](https://mlsanity.com/paper/2211.11869/) (2022)
- [An adaptive stochastic gradient-free approach for high-dimensional blackbox optimization](https://mlsanity.com/paper/2006.10887/) (2020)

## Abstract (arXiv)

In this effort, we consider the impact of regularization on the diversity of actions taken by policies generated from reinforcement learning agents trained using a policy gradient. Policy gradient agents are prone to entropy collapse, which means certain actions are seldomly, if ever, selected. We augment the optimization objective function for the policy with terms constructed from various varphi-divergences and Maximum Mean Discrepancy which encourages current policies to follow different state visitation and/or action choice distribution than previously computed policies. We provide numerical experiments using MNIST, CIFAR10, and Spotify datasets. The results demonstrate the advantage of diversity-promoting policy regularization and that its use on gradient-based approaches have significantly improved performance on a variety of personalization tasks. Furthermore, numerical evidence is given to show that policy regularization increases performance without losing accuracy.