mlsanityData to 7 Oct 2026

Large Language Models are Zero-Shot Rankers for Recommender Systems

arXiv 2305.08845 ↗First posted 15 May 2023cs.IR cs.CLcited by 307

Not on the map. Why?

Cited by 27 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, large language models (LLMs) (e.g., GPT-4) have demonstrated impressive general-purpose task-solving abilities, including the potential to approach recommendation tasks. Along this line of research, this work aims to investigate the capacity of LLMs that act as the ranking model for recommender systems. We first formalize the recommendation problem as a conditional ranking task, considering sequential interaction histories as conditions and the items retrieved by other candidate generation models as candidates. To solve the ranking task by LLMs, we carefully design the prompting template and conduct extensive experiments on two widely-used datasets. We show that LLMs have promising zero-shot ranking abilities but (1) struggle to perceive the order of historical interactions, and (2) can be biased by popularity or item positions in the prompts. We demonstrate that these issues can be alleviated using specially designed prompting and bootstrapping strategies. Equipped with these insights, zero-shot LLMs can even challenge conventional recommendation models when ranking candidates are retrieved by multiple candidate generators. The code and processed datasets are available at https://github.com/RUCAIBox/LLMRank.

Key facts. Cited by 307 papers in the map (data through 7 Oct 2026). Builds on 55 of its 99 references that are in the map. First posted on arXiv 15 May 2023.

Builds on

55 of 99 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2018Self-Attentive Sequential Recommendationcited by 1,169
  12. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,134
38 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  2. 2015Session-based Recommendations with Recurrent Neural Networkscited by 685
  3. 2018Personalized Top-N Sequential Recommendation via Convolutional Sequence Embeddingcited by 473
  4. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 396
  5. 2023TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendationcited by 336
  6. 2020S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximizationcited by 311
  7. 2023Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender Systemcited by 241
  8. 2020Bias and Debias in Recommender System: A Survey and Future Directionscited by 239
  9. 2023Is ChatGPT a Good Recommender? A Preliminary Studycited by 192
  10. 2020A Survey on Knowledge Graph-Based Recommender Systemscited by 170
  11. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  12. 2023Uncovering ChatGPT's Capabilities in Recommender Systemscited by 160
  13. 2022M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systemscited by 152
  14. 2020RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithmscited by 137
  15. 2022Towards Universal Sequence Representation Learning for Recommender Systemscited by 120
  16. 2023Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Predictioncited by 113
  17. 2023Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendationcited by 113
  18. 2022Learning Vector-Quantized Item Representation for Transferable Sequential Recommenderscited by 112
  19. 2023How to Index Item IDs for Recommendation Foundation Modelscited by 102
  20. 2023GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretationcited by 101
  21. 2022Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learningcited by 85
  22. 2022Personalized Prompt Learning for Explainable Recommendationcited by 79
  23. 2021Cross-Domain Recommendation: Challenges, Progress, and Prospectscited by 74
  24. 2016Item2Vec: Neural Item Embedding for Collaborative Filteringcited by 72
  25. 2017Translation-based Recommendationcited by 70
  26. 2023Generative Recommendation: Towards Next-generation Recommender Paradigmcited by 69
  27. 2023Zero-Shot Next-Item Recommendation using Large Pretrained Language Modelscited by 53
  28. 2023A Survey on Long Text Modeling with Transformerscited by 47
  29. 2021Zero-Shot Recommender Systemscited by 45
  30. 2023Pre-train, Prompt and Recommendation: A Comprehensive Survey of Language Modelling Paradigm Adaptations in Recommender Systemscited by 45
  31. 2020CATN: Cross-Domain Recommendation for Cold-Start Users via Aspect Transfer Networkcited by 42
  32. 2022RecBole 2.0: Towards a More Up-to-Date Recommendation Librarycited by 39
  33. 2022CORE: Simple and Effective Session-based Recommendation within Consistent Representation Spacecited by 37
  34. 2021Personalized Transfer of User Preferences for Cross-domain Recommendationcited by 34
  35. 2020One Person, One Model, One World: Learning Continual User Representation without Forgettingcited by 30
  36. 2021A Survey on Cross-domain Recommendation: Taxonomies, Methods, and Future Directionscited by 30
  37. 2021Scaling Law for Recommendation Models: Towards General-purpose User Representationscited by 26
  38. 2023Prompt Learning for News Recommendationcited by 25

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 42; 2024: 116; 2025: 83; 2026: 66.

307 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2023A Survey on Large Language Models for Recommendationcited by 378
  3. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  4. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  5. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  6. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  7. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  8. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  9. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  10. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 188
  11. 2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 184
  12. 2023Representation Learning with Large Language Models for Recommendationcited by 155
38 more
  1. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 141
  2. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 139
  3. 2023RecMind: Large Language Model Powered Agent For Recommendationcited by 129
  4. 2023Large Language Models as Zero-Shot Conversational Recommenderscited by 126
  5. 2023AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systemscited by 116
  6. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  7. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  8. 2023ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendationcited by 76
  9. 2023Large Language Models for Generative Recommendation: A Survey and Visionary Discussionscited by 72
  10. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 69
  11. 2023Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferencescited by 66
  12. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  13. 2024LitLLM: A Toolkit for Scientific Literature Reviewcited by 61
  14. 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 59
  15. 2024A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)cited by 58
  16. 2025On the Emergence of Position Bias in Transformerscited by 57
  17. 2023LlamaRec: Two-Stage Recommendation using Large Language Models for Rankingcited by 52
  18. 2024IDGenRec: LLM-RecSys Alignment with Textual ID Learningcited by 50
  19. 2024Don't Forget to Connect! Improving RAG with Graph-based Rerankingcited by 49
  20. 2025A Survey on LLM-powered Agents for Recommender Systemscited by 49
  21. 2023Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Modelscited by 48
  22. 2023Exploring Large Language Model for Graph Data Understanding in Online Job Recommendationscited by 45
  23. 2024Evaluating Large Language Models as Generative User Simulators for Conversational Recommendationcited by 45
  24. 2023ChatCounselor: A Large Language Models for Mental Health Supportcited by 44
  25. 2024eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Datacited by 44
  26. 2023Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generationcited by 42
  27. 2023RETA-LLM: A Retrieval-Augmented Large Language Model Toolkitcited by 40
  28. 2024LLM-Ensemble: Optimal Large Language Model Ensemble Method for E-commerce Product Attribute Value Extractioncited by 39
  29. 2024TravelAgent: An AI Assistant for Personalized Travel Planningcited by 38
  30. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  31. 2023Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Costcited by 33
  32. 2024Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendationcited by 33
  33. 2025PersonaX: A Recommendation Agent Oriented User Modeling Framework for Long Behavior Sequencecited by 33
  34. 2024Customizing Language Models with Instance-wise LoRA for Sequential Recommendationcited by 32
  35. 2025Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learningcited by 32
  36. 2025SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluationcited by 32
  37. 2023Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Modelscited by 30
  38. 2023Large Language Models for Intent-Driven Session Recommendationscited by 30

50 most cited of 307 All 307 (JSON)