mlsanityData to 7 Oct 2026

Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction

arXiv 2305.06474 ↗First posted 10 May 2023cs.IR cs.LGcited by 113

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated exceptional capabilities in generalizing to new tasks in a zero-shot or few-shot manner. However, the extent to which LLMs can comprehend user preferences based on their previous behavior remains an emerging and still unclear research question. Traditionally, Collaborative Filtering (CF) has been the most effective method for these tasks, predominantly relying on the extensive volume of rating data. In contrast, LLMs typically demand considerably less data while maintaining an exhaustive world knowledge about each item, such as movies or products. In this paper, we conduct a thorough examination of both CF and LLMs within the classic task of user rating prediction, which involves predicting a user's rating for a candidate item based on their past ratings. We investigate various LLMs in different sizes, ranging from 250M to 540B parameters and evaluate their performance in zero-shot, few-shot, and fine-tuning scenarios. We conduct comprehensive analysis to compare between LLMs and strong CF methods, and find that zero-shot LLMs lag behind traditional recommender models that have the access to user interaction data, indicating the importance of user interaction data. However, through fine-tuning, LLMs achieve comparable or even better performance with only a small fraction of the training data, demonstrating their potential through data efficiency.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

27 of 44 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
15 more
  1. 2018Self-Attentive Sequential Recommendationcited by 1,169
  2. 2017Neural Collaborative Filteringcited by 1,108
  3. 2015Session-based Recommendations with Recurrent Neural Networkscited by 685
  4. 2019Text Summarization with Pretrained Encoderscited by 553
  5. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 396
  6. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  7. 2023Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender Systemcited by 241
  8. 2023Is ChatGPT a Good Recommender? A Preliminary Studycited by 192
  9. 2022M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systemscited by 152
  10. 2021M6: A Chinese Multimodal Pretrainercited by 91
  11. 2023Zero-Shot Next-Item Recommendation using Large Pretrained Language Modelscited by 53
  12. 2019Towards Making the Most of BERT in Neural Machine Translationcited by 43
  13. 2020Learning-to-Rank with BERT in TF-Rankingcited by 36
  14. 2021Pre-trained Language Model based Ranking in Baidu Searchcited by 33
  15. 2022Take a Fresh Look at Recommender Systems from an Evaluation Standpointcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 43; 2025: 30; 2026: 14.

113 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Models for Recommendationcited by 378
  2. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  3. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  4. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 188
  5. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  6. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 141
  7. 2023Recommender AI Agent: Integrating Large Language Models for Interactive Recommendationscited by 130
  8. 2023RecMind: Large Language Model Powered Agent For Recommendationcited by 129
  9. 2023Large Language Models as Zero-Shot Conversational Recommenderscited by 126
  10. 2023AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systemscited by 116
  11. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  12. 2023ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendationcited by 76
38 more
  1. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 74
  2. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  3. 2023Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferencescited by 66
  4. 2024HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modelingcited by 63
  5. 2024A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)cited by 58
  6. 2023LlamaRec: Two-Stage Recommendation using Large Language Models for Rankingcited by 52
  7. 2023Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Modelscited by 48
  8. 2023Exploring Large Language Model for Graph Data Understanding in Online Job Recommendationscited by 45
  9. 2024Evaluating Large Language Models as Generative User Simulators for Conversational Recommendationcited by 45
  10. 2024CoRAL: Collaborative Retrieval-Augmented Large Language Models Improve Long-tail Recommendationcited by 44
  11. 2025Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalizationcited by 44
  12. 2023Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestioncited by 43
  13. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  14. 2024TravelAgent: An AI Assistant for Personalized Travel Planningcited by 38
  15. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  16. 2025PersonaX: A Recommendation Agent Oriented User Modeling Framework for Long Behavior Sequencecited by 33
  17. 2025SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluationcited by 32
  18. 2024AI PERSONA: Towards Life-long Personalization of LLMscited by 30
  19. 2023Learning Personalized Alignment for Evaluating Open-ended Text Generationcited by 29
  20. 2023ClickPrompt: CTR Models are Strong Prompt Generators for Adapting Language Models to CTR Predictioncited by 27
  21. 2024Rethinking Large Language Model Architectures for Sequential Recommendationscited by 27
  22. 2024Leveraging LLM Reasoning Enhances Personalized Recommender Systemscited by 26
  23. 2025A Survey of Personalization: From RAG to Agentcited by 26
  24. 2024Can Small Language Models be Good Reasoners for Sequential Recommendation?cited by 25
  25. 2024LLM-Enhanced User-Item Interactions: Leveraging Edge Information for Optimized Recommendationscited by 24
  26. 2024CALRec: Contrastive Alignment of Generative LLMs for Sequential Recommendationcited by 23
  27. 2023Evaluating ChatGPT as a Recommender System: A Rigorous Approachcited by 20
  28. 2024Stealthy Attack on Large Language Model based Recommendationcited by 19
  29. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
  30. 2025Spotlight Your Instructions: Instruction-following with Dynamic Attention Steeringcited by 18
  31. 2023Knowledge Plugins: Enhancing Large Language Models for Domain-Specific Recommendationscited by 16
  32. 2024Federated Recommendation via Hybrid Retrieval Augmented Generationcited by 16
  33. 2024Lifelong Personalized Low-Rank Adaptation of Large Language Models for Recommendationcited by 16
  34. 2025Exploring the Potential of LLMs as Personalized Assistants: Dataset, Evaluation, and Analysiscited by 16
  35. 2025PRIME: Large Language Model Personalization with Cognitive Dual-Memory and Personalized Thought Processcited by 16
  36. 2023''Fifty Shades of Bias'': Normative Ratings of Gender Bias in GPT Generated English Textcited by 15
  37. 2025Personalize Your LLM: Fake it then Align itcited by 15
  38. 2025Instant Personalized Large Language Model Adaptation via Hypernetworkcited by 15

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Can Large Language Models Understand Preferences in Personalized Recommendation?
  2. 2025Generative Large Recommendation Models: Emerging Trends in LLMs for Recommendation
  3. 2026Toward User Preference Alignment in LLM Recommendation via Explicit Context Feedback
  4. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs
  5. 2026Personalized Benchmarking: Evaluating LLMs by Individual Preferences
  6. 2024Towards Next-Generation LLM-based Recommender Systems: A Survey and Beyond
  7. 2026Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation
  8. 2024Large Language Models Enhanced Collaborative Filtering