# AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems

arXiv: https://arxiv.org/abs/2505.19623 · HTML: https://mlsanity.com/paper/2505.19623/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in LLM reasoning for recommendation, one of 17 directions in LLM reasoning.
- Builds on 46 of its 55 references that are in the map.
- First posted on arXiv 26 May 2025.

## Placed in

- LLM reasoning / [LLM reasoning for recommendation](https://mlsanity.com/lr/llm-reasoning-for-recommendation/)
- LLM agents and tool use / [Agentic recommender systems](https://mlsanity.com/agents/agentic-recommender-systems/)

## Builds on (most cited first, 30 of 46)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System](https://mlsanity.com/paper/2303.14524/) (2023)
- [MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents](https://mlsanity.com/paper/2503.01935/) (2025)
- [AgentSquare: Automatic LLM Agent Search in Modular Design Space](https://mlsanity.com/paper/2410.06153/) (2024)
- [Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs](https://mlsanity.com/paper/2502.09597/) (2025)
- [A Survey of Graph Neural Networks for Recommender Systems: Challenges, Methods, and Directions](https://mlsanity.com/paper/2109.12843/) (2021)
- [Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations](https://mlsanity.com/paper/2308.16505/) (2023)
- [RecMind: Large Language Model Powered Agent For Recommendation](https://mlsanity.com/paper/2308.14296/) (2023)
- [AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems](https://mlsanity.com/paper/2310.09233/) (2023)
- [On Generative Agents in Recommendation](https://mlsanity.com/paper/2310.10108/) (2023)

## Built on by (most cited first, 23 of 23)

- [Reason-to-Recommend: Using Interaction-of-Thought Reasoning to Enhance LLM Recommendation](https://mlsanity.com/paper/2506.05069/) (2025)
- [FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents](https://mlsanity.com/paper/2507.21071/) (2025)
- [RecoWorld: Building Simulated Environments for Agentic Recommender Systems](https://mlsanity.com/paper/2509.10397/) (2025)
- [Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directions](https://mlsanity.com/paper/2602.22680/) (2026)
- [RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation](https://mlsanity.com/paper/2605.20204/) (2026)
- [See, Think, Act: Online Shopper Behavior Simulation with VLM Agents](https://mlsanity.com/paper/2510.19245/) (2025)
- [Internalizing Multi-Agent Reasoning for Accurate and Efficient LLM-based Recommendation](https://mlsanity.com/paper/2602.09829/) (2026)
- [RecRM-Bench: Benchmarking Multidimensional Reward Modeling for Agentic Recommender Systems](https://mlsanity.com/paper/2605.11874/) (2026)
- [Autonomous Information Seeking: A Roadmap for Agentic Recommender Systems](https://mlsanity.com/paper/2607.04433/) (2026)
- [A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challenges](https://mlsanity.com/paper/2507.21117/) (2025)
- [τ-Rec: A Verifiable Benchmark for Agentic Recommender Systems](https://mlsanity.com/paper/2606.10156/) (2026)
- [ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Tests](https://mlsanity.com/paper/2510.26095/) (2025)
- [Position: Recommender Systems Should Move Beyond Platform-Centric Ranking toward Personal Agent-Mediated Recommendation](https://mlsanity.com/paper/2609.11942/) (2026)
- [To See or To Read: User Behavior Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2511.03845/) (2025)
- [RecoAtlas: From Semantic Plausibility to Set-Level Utility in LLM Recommendation Agents](https://mlsanity.com/paper/2605.18805/) (2026)
- [PaperFlow: Profiling, Recommending, and Adapting Across Daily Paper Streams](https://mlsanity.com/paper/2606.07454/) (2026)
- [APeB: Benchmarking Personalization Ability of Large Language Model Agents](https://mlsanity.com/paper/2607.03162/) (2026)
- [The User Asks, Platforms Compete: How Agentic Recommendation Markets Take Shape](https://mlsanity.com/paper/2607.25253/) (2026)
- [Attacking and Defending Multi-Agent Collaborative Filtering Systems Through Connectivity](https://mlsanity.com/paper/2608.03272/) (2026)
- [MAP: A Benchmark on Multimodal Accessibility Planning for Real World Places](https://mlsanity.com/paper/2608.28384/) (2026)
- [PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents](https://mlsanity.com/paper/2609.34930/) (2026)
- [ReMem: Rethinking Perception and Memory in Long-Context Recommendation Agents](https://mlsanity.com/paper/2609.37311/) (2026)
- [VirusCascade: Hijacking Collaborative Reflection in LLM-Powered Recommender Agents](https://mlsanity.com/paper/2609.38270/) (2026)

## Abstract (arXiv)

The emergence of agentic recommender systems powered by Large Language Models (LLMs) represents a paradigm shift in personalized recommendations, leveraging LLMs' advanced reasoning and role-playing capabilities to enable autonomous, adaptive decision-making. Unlike traditional recommendation approaches, agentic recommender systems can dynamically gather and interpret user-item interactions from complex environments, generating robust recommendation strategies that generalize across diverse scenarios. However, the field currently lacks standardized evaluation protocols to systematically assess these methods. To address this critical gap, we propose: (1) an interactive textual recommendation simulator incorporating rich user and item metadata and three typical evaluation scenarios (classic, evolving-interest, and cold-start recommendation tasks); (2) a unified modular framework for developing and studying agentic recommender systems; and (3) the first comprehensive benchmark comparing 10 classical and agentic recommendation methods. Our findings demonstrate the superiority of agentic systems and establish actionable design guidelines for their core components. The benchmark environment has been rigorously validated through an open challenge and remains publicly available with a continuously maintained leaderboard~footnote[2]https://tsinghua-fib-lab.github.io/AgentSocietyChallenge/pages/overview.html, fostering ongoing community engagement and reproducible research. The benchmark is available at: hyperlinkhttps://huggingface.co/datasets/SGJQovo/AgentRecBenchhttps://huggingface.co/datasets/SGJQovo/AgentRecBench.