# Choosing the Best of Both Worlds: Diverse and Novel Recommendations through Multi-Objective Reinforcement Learning

arXiv: https://arxiv.org/abs/2110.15097 · HTML: https://mlsanity.com/paper/2110.15097/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 49 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Recurrent Neural Networks with Top-k Gains for Session-based Recommendations](https://mlsanity.com/paper/1706.03847/) (2017)
- [A Simple Convolutional Generative Network for Next Item Recommendation](https://mlsanity.com/paper/1808.05163/) (2018)
- [Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems](https://mlsanity.com/paper/1902.05570/) (2019)
- [Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1610.02707/) (2016)
- [Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning](https://mlsanity.com/paper/1802.06501/) (2018)
- [Generative Adversarial User Model for Reinforcement Learning Based Recommendation System](https://mlsanity.com/paper/1812.10613/) (2018)
- [Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application](https://mlsanity.com/paper/1803.00710/) (2018)
- [Multi-Gradient Descent for Multi-Objective Recommender Systems](https://mlsanity.com/paper/2001.00846/) (2019)
- [Environment Reconstruction with Hidden Confounders for Reinforcement Learning based Recommendation](https://mlsanity.com/paper/1907.06584/) (2019)

## Built on by (most cited first, 14 of 14)

- [Two-Stage Constrained Actor-Critic for Short Video Recommendation](https://mlsanity.com/paper/2302.01680/) (2023)
- [Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2407.17466/) (2024)
- [Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2405.03082/) (2024)
- [Constrained Reinforcement Learning for Short Video Recommendation](https://mlsanity.com/paper/2205.13248/) (2022)
- [Optimizing Novelty of Top-k Recommendations using Large Language Models and Reinforcement Learning](https://mlsanity.com/paper/2406.14169/) (2024)
- [Post-hoc Selection of Pareto-Optimal Solutions in Search and Recommendation](https://mlsanity.com/paper/2306.12165/) (2023)
- [Model-enhanced Contrastive Reinforcement Learning for Sequential Recommendation](https://mlsanity.com/paper/2310.16566/) (2023)
- [MOReGIn: Multi-Objective Recommendation at the Global and Individual Levels](https://mlsanity.com/paper/2401.12593/) (2024)
- [Goal-Conditioned Supervised Learning for Multi-Objective Recommendation](https://mlsanity.com/paper/2412.08911/) (2024)
- [Breaking the Filter Bubble: A Semantic Pareto-DQN Framework for Multi-Objective Recommendation](https://mlsanity.com/paper/2606.24042/) (2026)
- [Aligning GPTRec with Beyond-Accuracy Goals with Reinforcement Learning](https://mlsanity.com/paper/2403.04875/) (2024)
- [Large Language Model-Enhanced Reinforcement Learning for Diverse and Novel Recommendations](https://mlsanity.com/paper/2507.21274/) (2025)
- [Breaking the Cold-Start Barrier: Reinforcement Learning with Double and Dueling DQNs](https://mlsanity.com/paper/2508.21259/) (2025)
- [Affective Music Recommendation: A Rollout-Based World Model for Offline Preference Optimization](https://mlsanity.com/paper/2605.28810/) (2026)

## Abstract (arXiv)

Since the inception of Recommender Systems (RS), the accuracy of the recommendations in terms of relevance has been the golden criterion for evaluating the quality of RS algorithms. However, by focusing on item relevance, one pays a significant price in terms of other important metrics: users get stuck in a "filter bubble" and their array of options is significantly reduced, hence degrading the quality of the user experience and leading to churn. Recommendation, and in particular session-based/sequential recommendation, is a complex task with multiple - and often conflicting objectives - that existing state-of-the-art approaches fail to address. In this work, we take on the aforementioned challenge and introduce Scalarized Multi-Objective Reinforcement Learning (SMORL) for the RS setting, a novel Reinforcement Learning (RL) framework that can effectively address multi-objective recommendation tasks. The proposed SMORL agent augments standard recommendation models with additional RL layers that enforce it to simultaneously satisfy three principal objectives: accuracy, diversity, and novelty of recommendations. We integrate this framework with four state-of-the-art session-based recommendation models and compare it with a single-objective RL agent that only focuses on accuracy. Our experimental results on two real-world datasets reveal a substantial increase in aggregate diversity, a moderate increase in accuracy, reduced repetitiveness of recommendations, and demonstrate the importance of reinforcing diversity and novelty as complementary objectives.