# Beyond Item Dissimilarities: Diversifying by Intent in Recommender Systems

arXiv: https://arxiv.org/abs/2405.12327 · HTML: https://mlsanity.com/paper/2405.12327/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 45 references that are in the map.
- First posted on arXiv 20 May 2024.

## Builds on (most cited first, 10 of 10)

- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [A Survey of Graph Neural Networks for Recommender Systems: Challenges, Methods, and Directions](https://mlsanity.com/paper/2109.12843/) (2021)
- [Deep Learning based Recommender System: A Survey and New Perspectives](https://mlsanity.com/paper/1707.07435/) (2017)
- [How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility](https://mlsanity.com/paper/1710.11214/) (2017)
- [Learning a Deep Listwise Context Model for Ranking Refinement](https://mlsanity.com/paper/1804.05936/) (2018)
- [Query expansion techniques for information retrieval: A survey](https://mlsanity.com/paper/1708.00247/) (2017)
- [Understanding Echo Chambers in E-commerce Recommender Systems](https://mlsanity.com/paper/2007.02474/) (2020)
- [Globally Optimized Mutual Influence Aware Ranking in E-Commerce Search](https://mlsanity.com/paper/1805.08524/) (2018)
- [Generic Intent Representation in Web Search](https://mlsanity.com/paper/1907.10710/) (2019)
- [Online Matching: A Real-time Bandit System for Large-scale Recommendations](https://mlsanity.com/paper/2307.15893/) (2023)

## Built on by (most cited first, 4 of 4)

- [Self-Evolving Recommendation System: End-To-End Autonomous Model Optimization With LLM Agents](https://mlsanity.com/paper/2602.10226/) (2026)
- [Beyond the Crowd: LLM-Augmented Community Notes for Governing Health Misinformation](https://mlsanity.com/paper/2510.11423/) (2025)
- [TransAct V2: Lifelong User Action Sequence Modeling on Pinterest Recommendation](https://mlsanity.com/paper/2506.02267/) (2025)
- [Adaptive Quality-Diversity Trade-offs for Large-Scale Batch Recommendation](https://mlsanity.com/paper/2602.02024/) (2026)

## Abstract (arXiv)

It has become increasingly clear that recommender systems that overly focus on short-term engagement prevents users from exploring diverse interests, ultimately hurting long-term user experience. To tackle this challenge, numerous diversification algorithms have been proposed. These algorithms typically rely on measures of item similarity, aiming to maximize the dissimilarity across items in the final set of recommendations. However, in this work, we demonstrate the benefits of going beyond item-level similarities by utilizing higher-level user understanding--specifically, user intents that persist across multiple interactions--in diversification. Our approach is motivated by the observation that user behaviors on online platforms are largely driven by their underlying intents. Therefore, recommendations should ensure that diverse user intents are accurately represented. While intent has primarily been studied in the context of search, it is less clear how to incorporate real-time dynamic intent predictions into recommender systems. To address this gap, we develop a probabilistic intent-based whole-page diversification framework for the final stage of a recommender system. Starting with a prior belief of user intents, the proposed framework sequentially selects items for each position based on these beliefs and subsequently updates posterior beliefs about the intents. This approach ensures that different user intents are represented on a page, towards optimizing long-term user experience. We experiment with the intent diversification framework on YouTube, the world's largest video recommendation platform, serving billions of users daily. Live experiments on a diverse set of intents show that the proposed framework increases Daily Active Users (DAU) and overall user enjoyment, validating its effectiveness in facilitating long-term planning.