Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testing
On the map: LLM agents and tool use → Agentic recommender systems
Abstract (arXiv)
In recommender systems, online A/B testing is a crucial method for evaluating the performance of different models. However, conducting online A/B testing often presents significant challenges, including substantial economic costs, user experience degradation, and considerable time requirements. With the Large Language Models' powerful capacity, LLM-based agent shows great potential to replace traditional online A/B testing. Nonetheless, current agents fail to simulate the perception process and interaction patterns, due to the lack of real environments and visual perception capability. To address these challenges, we introduce a multi-modal user agent for A/B testing (A/B Agent). Specifically, we construct a recommendation sandbox environment for A/B testing, enabling multimodal and multi-page interactions that align with real user behavior on online platforms. The designed agent leverages multimodal information perception, fine-grained user preferences, and integrates profiles, action memory retrieval, and a fatigue system to simulate complex human decision-making. We validated the potential of the agent as an alternative to traditional A/B testing from three perspectives: model, data, and features. Furthermore, we found that the data generated by A/B Agent can effectively enhance the capabilities of recommendation models. Our code is publicly available at https://github.com/Applied-Machine-Learning-Lab/ABAgent.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Agentic recommender systems, one of 16 directions in LLM agents and tool use. Builds on 37 of its 61 references that are in the map. First posted on arXiv 8 Jan 2026.
Builds on
37 of 61 references.
Closest ancestors · 1
Same direction, newest first
- 2023On Generative Agents in Recommendation16 Oct
Other references · 36
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
- 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
- 2016Wide & Deep Learning for Recommender Systemscited by 745
- 2024Understanding the planning of LLM agents: A surveycited by 414
- 2017Deep & Cross Network for Ad Click Predictionscited by 392
- 2018AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networkscited by 319
- 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
- 2017Neural Factorization Machines for Sparse Predictive Analyticscited by 183
- 2016Product-based Neural Networks for User Response Predictioncited by 145
- 2023User Behavior Simulation with Large Language Model based Agentscited by 136
- 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 99
- 2018Offline A/B testing for Recommender Systemscited by 94
24 more
- 2019RecSim: A Configurable Simulation Platform for Recommender Systemscited by 91
- 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 86
- 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
- 2018Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learningcited by 71
- 2018RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertisingcited by 65
- 2018Generative Adversarial User Model for Reinforcement Learning Based Recommendation Systemcited by 64
- 2020Open Benchmarking for Click-Through Rate Predictioncited by 53
- 2024Let Me Do It For You: Towards LLM Empowered Recommendation via Tool Learningcited by 51
- 2019Adaptive Factorization Network: Learning Adaptive-Order Feature Interactionscited by 48
- 2024NoteLLM-2: Multimodal Large Representation Models for Recommendationcited by 44
- 2025MMDocIR: Benchmarking Multimodal Retrieval for Long Documentscited by 43
- 2024LLM-Powered User Simulator for Recommender Systemcited by 38
- 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 33
- 2023KuaiSim: A Comprehensive Simulator for Recommender Systemscited by 30
- 2020Jointly Learning to Recommend and Advertisecited by 27
- 2024How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendationcited by 25
- 2024LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systemscited by 21
- 2022IntTower: the Next Generation of Two-Tower Model for Pre-Ranking Systemcited by 18
- 2023Exploration and Regularization of the Latent Action Space in Recommendationcited by 17
- 2023A Unified Framework for Multi-Domain CTR Prediction via Large Language Modelscited by 16
- 2021A/B Testing for Recommender Systems in a Two-sided Marketplacecited by 11
- 2024Behavior Alignment: A New Perspective of Evaluating LLM-based Conversational Recommender Systemscited by 11
- 2024SUBER: An RL Environment with Simulated Human Behavior for Recommender Systemscited by 11
- 2025Personalize Before Retrieve: LLM-based Personalized Query Expansion for User-Centric Retrievalcited by 11
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation
- 2025Towards Agentic Recommender Systems in the Era of Multimodal Large Language Models
- 2025AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems
- 2025Beyond Static Testbeds: An Interaction-Centric Agent Simulation Platform for Dynamic Recommender Systems
- 2026Automatic multimodal UX improvement recommendations from LLM agent user simulations
- 2023AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems
- 2025RecoWorld: Building Simulated Environments for Agentic Recommender Systems
- 2025AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agents