mlsanityData to 7 Oct 2026

Uncovering ChatGPT's Capabilities in Recommender Systems

arXiv 2305.02182 ↗First posted 3 May 2023cs.IRcited by 160

Not on the map. Why?

Abstract (arXiv)

The debut of ChatGPT has recently attracted the attention of the natural language processing (NLP) community and beyond. Existing studies have demonstrated that ChatGPT shows significant improvement in a range of downstream NLP tasks, but the capabilities and limitations of ChatGPT in terms of recommendations remain unclear. In this study, we aim to conduct an empirical analysis of ChatGPT's recommendation ability from an Information Retrieval (IR) perspective, including point-wise, pair-wise, and list-wise ranking. To achieve this goal, we re-formulate the above three recommendation policies into a domain-specific prompt format. Through extensive experiments on four datasets from different domains, we demonstrate that ChatGPT outperforms other large language models across all three ranking policies. Based on the analysis of unit cost improvements, we identify that ChatGPT with list-wise ranking achieves the best trade-off between cost and performance compared to point-wise and pair-wise ranking. Moreover, ChatGPT shows the potential for mitigating the cold start problem and explainable recommendation. To facilitate further explorations in this area, the full code and detailed original results are open-sourced at https://github.com/rainym00d/LLM4RS.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 May 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 64; 2025: 50; 2026: 21.

160 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2023A Survey on Large Language Models for Recommendationcited by 378
  3. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  4. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  6. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  7. 2024Tool Learning with Large Language Models: A Surveycited by 221
  8. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  9. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  10. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  11. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 188
  12. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
38 more
  1. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 141
  2. 2023Recommender AI Agent: Integrating Large Language Models for Interactive Recommendationscited by 130
  3. 2023AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systemscited by 116
  4. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  5. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  6. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  7. 2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
  8. 2023ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendationcited by 76
  9. 2023Large Language Models for Generative Recommendation: A Survey and Visionary Discussionscited by 72
  10. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  11. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  12. 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 59
  13. 2024A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)cited by 58
  14. 2023ONCE: Boosting Content-based Recommendation with Both Open- and Closed-source Large Language Modelscited by 54
  15. 2024IDGenRec: LLM-RecSys Alignment with Textual ID Learningcited by 50
  16. 2023Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Modelscited by 48
  17. 2023Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestioncited by 43
  18. 2023E4SRec: An Elegant Effective Efficient Extensible Solution of Large Language Models for Sequential Recommendationcited by 39
  19. 2024TravelAgent: An AI Assistant for Personalized Travel Planningcited by 38
  20. 2023Neural Retrievers are Biased Towards LLM-Generated Contentcited by 37
  21. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  22. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  23. 2025PersonaX: A Recommendation Agent Oriented User Modeling Framework for Long Behavior Sequencecited by 33
  24. 2024Customizing Language Models with Instance-wise LoRA for Sequential Recommendationcited by 32
  25. 2023A Large Language Model Enhanced Conversational Recommender Systemcited by 31
  26. 2023Large Language Models for Intent-Driven Session Recommendationscited by 30
  27. 2023Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBenchcited by 28
  28. 2024Towards Completeness-Oriented Tool Retrieval for Large Language Modelscited by 26
  29. 2024Towards Next-Generation LLM-based Recommender Systems: A Survey and Beyondcited by 26
  30. 2024Can Small Language Models be Good Reasoners for Sequential Recommendation?cited by 25
  31. 2024How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendationcited by 25
  32. 2024LLM-Enhanced User-Item Interactions: Leveraging Edge Information for Optimized Recommendationscited by 24
  33. 2025Retrieval Augmented Generation with Collaborative Filtering for Personalized Text Generationcited by 23
  34. 2024MMREC: LLM Based Multi-Modal Recommender Systemcited by 22
  35. 2023Evaluating ChatGPT as a Recommender System: A Rigorous Approachcited by 20
  36. 2024ChatDiet: Empowering Personalized Nutrition-Oriented Food Recommender Chatbots through an LLM-Augmented Frameworkcited by 20
  37. 2024Enhancing Recommendation Diversity by Re-ranking with Large Language Modelscited by 19
  38. 2024Stealthy Attack on Large Language Model based Recommendationcited by 19

50 most cited of 160 All 160 (JSON)