# SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation

arXiv: https://arxiv.org/abs/2504.12722 · HTML: https://mlsanity.com/paper/2504.12722/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 17 Apr 2025.

## Builds on (most cited first, 18 of 18)

- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018)
- [Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations](https://mlsanity.com/paper/2308.16505/) (2023)
- [Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks](https://mlsanity.com/paper/1708.04617/) (2017)
- [Large Language Models as Zero-Shot Conversational Recommenders](https://mlsanity.com/paper/2308.10053/) (2023)
- [Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction](https://mlsanity.com/paper/2305.06474/) (2023)
- [AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems](https://mlsanity.com/paper/2310.09233/) (2023)
- [GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretation](https://mlsanity.com/paper/2304.03879/) (2023)
- [Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems](https://mlsanity.com/paper/2002.09102/) (2020)
- [Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation](https://mlsanity.com/paper/2403.09738/) (2024)
- [Measuring the Business Value of Recommender Systems](https://mlsanity.com/paper/1908.08328/) (2019)
- [Tenrec: A Large-scale Multipurpose Benchmark Dataset for Recommender Systems](https://mlsanity.com/paper/2210.10629/) (2022)
- [KuaiSim: A Comprehensive Simulator for Recommender Systems](https://mlsanity.com/paper/2309.12645/) (2023)
- [LLM-Enhanced User-Item Interactions: Leveraging Edge Information for Optimized Recommendations](https://mlsanity.com/paper/2402.09617/) (2024)
- [Prospect Personalized Recommendation on Large Language Model-based Agent Platform](https://mlsanity.com/paper/2402.18240/) (2024)
- [Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models](https://mlsanity.com/paper/2402.08670/) (2024)

## Built on by (most cited first, 30 of 33)

- [Mind the Sim2Real Gap in User Simulation for Agentic Tasks](https://mlsanity.com/paper/2603.11245/) (2026)
- [CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulation](https://mlsanity.com/paper/2506.21805/) (2025)
- [RecoWorld: Building Simulated Environments for Agentic Recommender Systems](https://mlsanity.com/paper/2509.10397/) (2025)
- [RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation](https://mlsanity.com/paper/2605.20204/) (2026)
- [Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces](https://mlsanity.com/paper/2604.08362/) (2026)
- [MirrorBench: A Benchmark to Evaluate Conversational User-Proxy Agents for Human-Likeness](https://mlsanity.com/paper/2601.08118/) (2026)
- [SAGE: A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn AGent Evaluation](https://mlsanity.com/paper/2510.11997/) (2025)
- [AlignUSER: Human-Aligned LLM Agents via World Models for Recommender System Evaluation](https://mlsanity.com/paper/2601.00930/) (2026)
- [Beyond Static Testbeds: An Interaction-Centric Agent Simulation Platform for Dynamic Recommender Systems](https://mlsanity.com/paper/2505.16429/) (2025)
- [Agentic Recommender System with Hierarchical Belief-State Memory](https://mlsanity.com/paper/2605.14401/) (2026)
- [HyTRec: A Hybrid Temporal-Aware Attention Architecture for Long Behavior Sequential Recommendation](https://mlsanity.com/paper/2602.18283/) (2026)
- [Reinforcing Human Behavior Simulation via Verbal Feedback](https://mlsanity.com/paper/2605.20506/) (2026)
- [DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings](https://mlsanity.com/paper/2606.22877/) (2026)
- [GGBond: Growing Graph-Based AI-Agent Society for Socially-Aware Recommender Simulation](https://mlsanity.com/paper/2505.21154/) (2025)
- [A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challenges](https://mlsanity.com/paper/2507.21117/) (2025)
- [Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation](https://mlsanity.com/paper/2604.09549/) (2026)
- [ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluation](https://mlsanity.com/paper/2507.12674/) (2025)
- [Breaking User-Centric Agency: A Tri-Party Framework for Agent-Based Recommendation](https://mlsanity.com/paper/2603.10673/) (2026)
- [How Well Do Large Language Models Capture Human Personality?](https://mlsanity.com/paper/2606.18263/) (2026)
- [CORAL: An LLM-Native Harness for Production Recommender Systems](https://mlsanity.com/paper/2609.02730/) (2026)
- [ShiJianBench: From Dialogue to Decision for Long-Horizon Evaluation of Investment Advisors](https://mlsanity.com/paper/2608.01204/) (2026)
- [Learning from Online User Feedback for Shopping Agents](https://mlsanity.com/paper/2608.11604/) (2026)
- [PersonaAct: Simulating Short-Video Users with Personalized Agents for Counterfactual Filter Bubble Auditing](https://mlsanity.com/paper/2601.22547/) (2026)
- [NextAds: Towards Next-generation Personalized Video Advertising](https://mlsanity.com/paper/2603.02137/) (2026)
- [Through Their Eyes: Fixation-aligned Tuning for Personalized User Emulation](https://mlsanity.com/paper/2604.09368/) (2026)
- [PERCEIVE: A Benchmark for Personalized Emotion and Communication Behavior Understanding on Social Media](https://mlsanity.com/paper/2605.12525/) (2026)
- [SR-Agent: An Experience-Driven Agentic Framework for Post-Ranking Strategy Refinement in E-Commerce Recommendation](https://mlsanity.com/paper/2607.17719/) (2026)
- [LLMs struggle to simulate human belief updates in controlled environments](https://mlsanity.com/paper/2607.28347/) (2026)
- [CityReal: Human-Aligned Urban Behavior and City Dynamics Simulation with Large-Scale LLM Agents](https://mlsanity.com/paper/2608.16897/) (2026)
- [Ask or Answer: A Decision Framework for Multi-Turn Health Misinformation Intervention](https://mlsanity.com/paper/2608.21721/) (2026)

## Abstract (arXiv)

Recommender systems play a central role in numerous real-life applications, yet evaluating their performance remains a significant challenge due to the gap between offline metrics and online behaviors. Given the scarcity and limits (e.g., privacy issues) of real user data, we introduce SimUSER, an agent framework that serves as believable and cost-effective human proxies. SimUSER first identifies self-consistent personas from historical data, enriching user profiles with unique backgrounds and personalities. Then, central to this evaluation are users equipped with persona, memory, perception, and brain modules, engaging in interactions with the recommender system. SimUSER exhibits closer alignment with genuine humans than prior work, both at micro and macro levels. Additionally, we conduct insightful experiments to explore the effects of thumbnails on click rates, the exposure effect, and the impact of reviews on user engagement. Finally, we refine recommender system parameters based on offline A/B test results, resulting in improved user engagement in the real world.