# Enhancing Recommendation Diversity by Re-ranking with Large Language Models

arXiv: https://arxiv.org/abs/2401.11506 · HTML: https://mlsanity.com/paper/2401.11506/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 79 references that are in the map.
- First posted on arXiv 21 Jan 2024.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation](https://mlsanity.com/paper/2305.00447/) (2023)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Language Models are Realistic Tabular Data Generators](https://mlsanity.com/paper/2210.06280/) (2022)
- [Recommender Systems in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2307.02046/) (2023)
- [Uncovering ChatGPT's Capabilities in Recommender Systems](https://mlsanity.com/paper/2305.02182/) (2023)
- [M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems](https://mlsanity.com/paper/2205.08084/) (2022)
- [Text Is All You Need: Learning Language Representations for Sequential Recommendation](https://mlsanity.com/paper/2305.13731/) (2023)
- [How Can Recommender Systems Benefit from Large Language Models: A Survey](https://mlsanity.com/paper/2306.05817/) (2023)
- [Towards Universal Sequence Representation Learning for Recommender Systems](https://mlsanity.com/paper/2206.05941/) (2022)
- [Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders](https://mlsanity.com/paper/2210.12316/) (2022)
- [GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretation](https://mlsanity.com/paper/2304.03879/) (2023)
- [A Survey on the Fairness of Recommender Systems](https://mlsanity.com/paper/2206.03761/) (2022)
- [Personalized Prompt Learning for Explainable Recommendation](https://mlsanity.com/paper/2202.07371/) (2022)
- [Zero-Shot Recommender Systems](https://mlsanity.com/paper/2105.08318/) (2021)
- [A Survey on Model Compression and Acceleration for Pretrained Language Models](https://mlsanity.com/paper/2202.07105/) (2022)
- [TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter](https://mlsanity.com/paper/2209.07562/) (2022)
- [Pre-trained Language Model for Web-scale Retrieval in Baidu Search](https://mlsanity.com/paper/2106.03373/) (2021)
- [Large Language Model Augmented Narrative Driven Recommendations](https://mlsanity.com/paper/2306.02250/) (2023)

## Built on by (most cited first, 19 of 19)

- [Towards Next-Generation LLM-based Recommender Systems: A Survey and Beyond](https://mlsanity.com/paper/2410.19744/) (2024)
- [Exploring the Impact of Large Language Models on Recommender Systems: An Extensive Review](https://mlsanity.com/paper/2402.18590/) (2024)
- [Improving LLM-powered Recommendations with Personalized Information](https://mlsanity.com/paper/2502.13845/) (2025)
- [In-context Ranking Preference Optimization](https://mlsanity.com/paper/2504.15477/) (2025)
- [Prompt-to-Slate: Diffusion Models for Prompt-Conditioned Slate Generation](https://mlsanity.com/paper/2408.06883/) (2024)
- [Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification](https://mlsanity.com/paper/2603.03336/) (2026)
- [Efficient Personalized Reranking with Semi-Autoregressive Generation and Online Knowledge Distillation](https://mlsanity.com/paper/2603.07107/) (2026)
- [MUSS: Multilevel Subset Selection for Relevance and Diversity](https://mlsanity.com/paper/2503.11126/) (2025)
- [Echoes in Filter Bubble: Diagnosing and Curing Popularity Bias in Generative Recommenders](https://mlsanity.com/paper/2605.16825/) (2026)
- [Large Language Models, and LLM-Based Agents, Should Be Used to Enhance the Digital Public Sphere](https://mlsanity.com/paper/2410.12123/) (2024)
- [LLM-Enhanced Reranking for Complementary Product Recommendation](https://mlsanity.com/paper/2507.16237/) (2025)
- [PCN-Rec: Agentic Proof-Carrying Negotiation for Reliable Governance-Constrained Recommendation](https://mlsanity.com/paper/2601.09771/) (2026)
- [LiveGraph: Active-Structure Neural Re-ranking for Exercise Recommendation](https://mlsanity.com/paper/2602.17036/) (2026)
- [Solving the Content Gap in Roblox Game Recommendations: LLM-Based Profile Generation and Reranking](https://mlsanity.com/paper/2502.06802/) (2025)
- [Enhancing Learned Knowledge in LoRA Adapters Through Efficient Contrastive Decoding on Ascend NPUs](https://mlsanity.com/paper/2505.14620/) (2025)
- [Exploring Diversity, Novelty, and Popularity Bias in ChatGPT's Recommendations](https://mlsanity.com/paper/2601.01997/) (2026)
- [Ideological Isolation in Online Social Networks: A Survey of Computational Definitions, Metrics, and Mitigation Strategies](https://mlsanity.com/paper/2601.07884/) (2026)
- [Adaptive Quality-Diversity Trade-offs for Large-Scale Batch Recommendation](https://mlsanity.com/paper/2602.02024/) (2026)
- [LLM-Assisted Reranking to Operationalize Nuanced Objectives in Recommender Systems](https://mlsanity.com/paper/2606.02883/) (2026)

## Abstract (arXiv)

It has long been recognized that it is not enough for a Recommender System (RS) to provide recommendations based only on their relevance to users. Among many other criteria, the set of recommendations may need to be diverse. Diversity is one way of handling recommendation uncertainty and ensuring that recommendations offer users a meaningful choice. The literature reports many ways of measuring diversity and improving the diversity of a set of recommendations, most notably by re-ranking and selecting from a larger set of candidate recommendations. Driven by promising insights from the literature on how to incorporate versatile Large Language Models (LLMs) into the RS pipeline, in this paper we show how LLMs can be used for diversity re-ranking. We begin with an informal study that verifies that LLMs can be used for re-ranking tasks and do have some understanding of the concept of item diversity. Then, we design a more rigorous methodology where LLMs are prompted to generate a diverse ranking from a candidate ranking using various prompt templates with different re-ranking instructions in a zero-shot fashion. We conduct comprehensive experiments testing state-of-the-art LLMs from the GPT and Llama families. We compare their re-ranking capabilities with random re-ranking and various traditional re-ranking methods from the literature. We open-source the code of our experiments for reproducibility. Our findings suggest that the trade-offs (in terms of performance and costs, among others) of LLM-based re-rankers are superior to those of random re-rankers but, as yet, inferior to the ones of traditional re-rankers. However, the LLM approach is promising. LLMs exhibit improved performance on many natural language processing and recommendation tasks and lower inference costs. Given these trends, we can expect LLM-based re-ranking to become more competitive soon.