mlsanityData to 7 Oct 2026

Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation

arXiv 2604.09549 ↗First posted 26 Jan 2026cs.IR cs.AIcited by 3

On the map: LLM agents and tool use → Agentic recommender systems

Abstract (arXiv)

Recommender systems are central to online services, enabling users to navigate through massive amounts of content across various domains. However, their evaluation remains challenging due to the disconnect between offline metrics and online performance. The emergence of Large Language Model-powered agents offers a promising solution, yet existing studies model users in isolation, neglecting the contextual factors such as time, location, and needs, which fundamentally shape human decision-making. In this paper, we introduce ContextSim, an LLM agent framework that simulates believable user proxies by anchoring interactions in daily life activities. Namely, a life simulation module generates scenarios specifying when, where, and why users engage with recommendations. To align preferences with genuine humans, we model agents' internal thoughts and enforce consistency at both the action and trajectory levels. Experiments across domains show our method generates interactions more closely aligned with human behavior than prior work. We further validate our approach through offline A/B testing correlation and show that RS parameters optimized using ContextSim yield improved real-world engagement.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Agentic recommender systems, one of 16 directions in LLM agents and tool use. Builds on 20 of its 34 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

20 of 34 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026AlignUSER: Human-Aligned LLM Agents via World Models for Recommender System Evaluation2 Jan
  2. 2025Beyond Static Testbeds: An Interaction-Centric Agent Simulation Platform for Dynamic Recommender Systems22 May
  3. 2024Prospect Personalized Recommendation on Large Language Model-based Agent Platform28 Feb
  4. 2023AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems13 Oct
  5. 2023Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations31 Aug
  6. 2023RecMind: Large Language Model Powered Agent For Recommendation28 Aug

Other references · 14

  1. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  2. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  3. 2017Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networkscited by 126
  4. 2023Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Predictioncited by 113
  5. 2023GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretationcited by 101
  6. 2020Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systemscited by 98
  7. 2024Evaluating Large Language Models as Generative User Simulators for Conversational Recommendationcited by 45
  8. 2019Measuring the Business Value of Recommender Systemscited by 32
  9. 2025SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluationcited by 32
  10. 2023KuaiSim: A Comprehensive Simulator for Recommender Systemscited by 30
  11. 2025CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulationcited by 28
  12. 2024LLM-Enhanced User-Item Interactions: Leveraging Edge Information for Optimized Recommendationscited by 24
2 more
  1. 2025Simulating Society Requires Simulating Thoughtcited by 12
  2. 2019Quantitative analysis of Matthew effect and sparsity problem of recommender systemscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluationcited by 1
  2. 2026Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentationcited by 0
  3. 2026CityReal: Human-Aligned Urban Behavior and City Dynamics Simulation with Large-Scale LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testing
  2. 2025AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems
  3. 2025Temporal-Aware User Behaviour Simulation with Large Language Models for Recommender Systems
  4. 2025RecUserSim: A Realistic and Diverse User Simulator for Evaluating Conversational Recommender Systems
  5. 2023On Generative Agents in Recommendation
  6. 2025RecoWorld: Building Simulated Environments for Agentic Recommender Systems
  7. 2026Who Are We Recommending To? Recommender Systems in the Agentic Web
  8. 2025Towards Agentic Recommender Systems in the Era of Multimodal Large Language Models