# Towards Next-Generation Recommender Systems: A Benchmark for Personalized Recommendation Assistant with LLMs

arXiv: https://arxiv.org/abs/2503.09382 · HTML: https://mlsanity.com/paper/2503.09382/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Builds on (most cited first, 20 of 20)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Graph Neural Networks for Social Recommendation](https://mlsanity.com/paper/1902.07243/) (2019)
- [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System](https://mlsanity.com/paper/2303.14524/) (2023)
- [Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation](https://mlsanity.com/paper/2305.07609/) (2023)
- [CoLLM: Integrating Collaborative Embeddings into Large Language Models for Recommendation](https://mlsanity.com/paper/2310.19488/) (2023)
- [CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs](https://mlsanity.com/paper/2401.11314/) (2024)
- [Knowledge Graph Retrieval-Augmented Generation for LLM-based Recommendation](https://mlsanity.com/paper/2501.02226/) (2025)
- [LLaRA: Large Language-Recommendation Assistant](https://mlsanity.com/paper/2312.02445/) (2023)
- [LLMRec: Benchmarking Large Language Models on Recommendation Task](https://mlsanity.com/paper/2308.12241/) (2023)
- [MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation](https://mlsanity.com/paper/2502.11903/) (2025)
- [Knowledge Graph Completion Models are Few-shot Learners: An Empirical Study of Relation Labeling in E-commerce with LLMs](https://mlsanity.com/paper/2305.09858/) (2023)
- [Can Large Language Models Understand Preferences in Personalized Recommendation?](https://mlsanity.com/paper/2501.13391/) (2025)
- [Lessons from Building StackSpot AI: A Contextualized AI Coding Assistant](https://mlsanity.com/paper/2311.18450/) (2023)

## Built on by (most cited first, 21 of 21)

- [A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models](https://mlsanity.com/paper/2503.23350/) (2025)
- [Interactive Recommendation Agent with Active User Commands](https://mlsanity.com/paper/2509.21317/) (2025)
- [Diffusion Generative Recommendation with Continuous Tokens](https://mlsanity.com/paper/2504.12007/) (2025)
- [MR.Rec: Synergizing Memory and Reasoning for Personalized Recommendation Assistant with LLMs](https://mlsanity.com/paper/2510.14629/) (2025)
- [RecRM-Bench: Benchmarking Multidimensional Reward Modeling for Agentic Recommender Systems](https://mlsanity.com/paper/2605.11874/) (2026)
- [URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2603.19281/) (2026)
- [GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification](https://mlsanity.com/paper/2603.29112/) (2026)
- [ReRec: Reasoning-Augmented LLM-based Recommendation Assistant via Reinforcement Fine-tuning](https://mlsanity.com/paper/2604.07851/) (2026)
- [Your "Pro" LLM Subscription May Actually Be "Free": Exposing Fingerprint Spoofing Risks in LLM Inference Services](https://mlsanity.com/paper/2606.16100/) (2026)
- [AdaptFuse: Training-Free Sequential Preference Inference via Externalized Bayesian Computation](https://mlsanity.com/paper/2604.03925/) (2026)
- [Mixture-of-Experts Knowledge Graph Retrieval-Augmented Generation for Multi-Agent LLM-based Recommendation](https://mlsanity.com/paper/2605.28175/) (2026)
- [Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2604.15301/) (2026)
- [Psi-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues](https://mlsanity.com/paper/2606.02754/) (2026)
- [Diffusion Language Model for Recommendation](https://mlsanity.com/paper/2607.21519/) (2026)
- [UserToolBench: A User-Profile-Hidden Benchmark for Personalized Decision Making in Tool-Use LLMs](https://mlsanity.com/paper/2608.10042/) (2026)
- [RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation](https://mlsanity.com/paper/2609.00918/) (2026)
- [AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalization](https://mlsanity.com/paper/2609.19793/) (2026)
- [PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents](https://mlsanity.com/paper/2609.34930/) (2026)
- [ReMem: Rethinking Perception and Memory in Long-Context Recommendation Agents](https://mlsanity.com/paper/2609.37311/) (2026)
- [RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce](https://mlsanity.com/paper/2609.38974/) (2026)
- [Does Every User Need a Private LoRA? Decoupling Personalization from Per-User Adaptation](https://mlsanity.com/paper/2610.02353/) (2026)

## Abstract (arXiv)

Recommender systems (RecSys) are widely used across various modern digital platforms and have garnered significant attention. Traditional recommender systems usually focus only on fixed and simple recommendation scenarios, making it difficult to generalize to new and unseen recommendation tasks in an interactive paradigm. Recently, the advancement of large language models (LLMs) has revolutionized the foundational architecture of RecSys, driving their evolution into more intelligent and interactive personalized recommendation assistants. However, most existing studies rely on fixed task-specific prompt templates to generate recommendations and evaluate the performance of personalized assistants, which limits the comprehensive assessments of their capabilities. This is because commonly used datasets lack high-quality textual user queries that reflect real-world recommendation scenarios, making them unsuitable for evaluating LLM-based personalized recommendation assistants. To address this gap, we introduce RecBench+, a new dataset benchmark designed to access LLMs' ability to handle intricate user recommendation needs in the era of LLMs. RecBench+ encompasses a diverse set of queries that span both hard conditions and soft preferences, with varying difficulty levels. We evaluated commonly used LLMs on RecBench+ and uncovered below findings: 1) LLMs demonstrate preliminary abilities to act as recommendation assistants, 2) LLMs are better at handling queries with explicitly stated conditions, while facing challenges with queries that require reasoning or contain misleading information. Our dataset has been released at https://github.com/jiani-huang/RecBench.git.