mlsanityData to 14 Oct 2026

Do LLMs Understand Collaborative Signals? Diagnosis and Repair

arXiv 2505.20730 ↗First posted 27 May 2025cs.IR cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Collaborative information from user-item interactions is a fundamental source of signal in successful recommender systems. Recently, researchers have attempted to incorporate this knowledge into large language model-based recommender approaches (LLMRec) to enhance their performance. However, there has been little fundamental analysis of whether LLMs can effectively reason over collaborative information. In this paper, we analyze the ability of LLMs to reason about collaborative information in recommendation tasks, comparing their performance to traditional matrix factorization (MF) models. We propose a simple and effective method to improve LLMs' reasoning capabilities using retrieval-augmented generation (RAG) over the user-item interaction matrix with four different prompting strategies. Our results show that the LLM outperforms the MF model whenever we provide relevant information in a clear and easy-to-follow format, and prompt the LLM to reason based on it. We observe that with this strategy, in almost all cases, the more information we provide, the better the LLM performs.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 31 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

26 of 31 references.

References · 26

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  4. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  5. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  6. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  7. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  8. 2023A Survey on Large Language Models for Recommendationcited by 379
  9. 2023TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendationcited by 337
  10. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  11. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 189
  12. 2023LLMRec: Large Language Models with Graph Augmentation for Recommendationcited by 172
14 more
  1. 2023Uncovering ChatGPT's Capabilities in Recommender Systemscited by 161
  2. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
  3. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 142
  4. 2023Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendationcited by 114
  5. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  6. 2023Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferencescited by 66
  7. 2023Leveraging GPT-4 for Automatic Translation Post-Editingcited by 56
  8. 2023LLaRA: Large Language-Recommendation Assistantcited by 56
  9. 2023LlamaRec: Two-Stage Recommendation using Large Language Models for Rankingcited by 53
  10. 2024Large Language Models meet Collaborative Filtering: An Efficient All-round LLM-based Recommender Systemcited by 50
  11. 2023Text Summarization Using Large Language Models: A Comparative Study of MPT-7b-instruct, Falcon-7b-instruct, and OpenAI Chat-GPT Modelscited by 36
  12. 2024Visualization Generation with Large Language Models: An Evaluationcited by 29
  13. 2024Large Language Models Enhanced Collaborative Filteringcited by 27
  14. 2024Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Studycited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Do Evidence-Reading Diagnostics Improve Interface Selection in Small LLM Recommenders?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CoRAL: Collaborative Retrieval-Augmented Large Language Models Improve Long-tail Recommendation
  2. 2025LLMs as Better Recommenders with Natural Language Collaborative Signals: A Self-Assessing Retrieval Approach
  3. 2025Retrieval-Augmented Purifier for Robust LLM-Empowered Recommendation
  4. 2025Towards Comprehensible Recommendation with Large Language Model Fine-tuning ↗
  5. 2026RRCM: Ranking-Driven Retrieval over Collaborative and Meta Memories for LLM Recommendation
  6. 2025ItemRAG: Item-Based Retrieval-Augmented Generation for LLM-Based Recommendation
  7. 2025Catalog-Native LLM: Speaking Item-ID Dialect with Less Entanglement for Recommendation
  8. 2025Reason-to-Recommend: Using Interaction-of-Thought Reasoning to Enhance LLM Recommendation