# Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping

arXiv: https://arxiv.org/abs/2510.07230 · HTML: https://mlsanity.com/paper/2510.07230/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization](https://mlsanity.com/paper/2503.12937/) (2025)
- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models](https://mlsanity.com/paper/2504.11468/) (2025)
- [WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2505.16421/) (2025)
- [User Behavior Simulation with Large Language Model based Agents](https://mlsanity.com/paper/2306.02552/) (2023)
- [Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinement](https://mlsanity.com/paper/2402.11060/) (2024)
- [UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design](https://mlsanity.com/paper/2502.12561/) (2025)
- [Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learning](https://mlsanity.com/paper/2503.24289/) (2025)
- [OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation](https://mlsanity.com/paper/2506.05606/) (2025)
- [Simulating Cooperative Prosocial Behavior with Multi-Agent LLMs: Evidence and Mechanisms for AI Agents to Inform Policy Decisions](https://mlsanity.com/paper/2502.12504/) (2025)
- [AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agents](https://mlsanity.com/paper/2504.09723/) (2025)
- [Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning](https://mlsanity.com/paper/2507.17842/) (2025)

## Built on by (most cited first, 19 of 19)

- [Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents](https://mlsanity.com/paper/2601.20144/) (2026)
- [ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents](https://mlsanity.com/paper/2605.16116/) (2026)
- [FileGram: Grounding Agent Personalization in File-System Behavioral Traces](https://mlsanity.com/paper/2604.04901/) (2026)
- [Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization](https://mlsanity.com/paper/2602.11351/) (2026)
- [What Makes a Sale? Simulating End-to-End Seller--Buyer Retail Dynamics with LLM Agents](https://mlsanity.com/paper/2604.04468/) (2026)
- [SimPersona: Learning Discrete Buyer Personas from Raw Clickstreams for Grounded E-Commerce Agents](https://mlsanity.com/paper/2605.14205/) (2026)
- [Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench](https://mlsanity.com/paper/2605.17079/) (2026)
- [SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents](https://mlsanity.com/paper/2605.19219/) (2026)
- [Large Behavior Model: A Promptable Digital Twin of the Retail Customer](https://mlsanity.com/paper/2607.06993/) (2026)
- [PersonaAct: Simulating Short-Video Users with Personalized Agents for Counterfactual Filter Bubble Auditing](https://mlsanity.com/paper/2601.22547/) (2026)
- [Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark](https://mlsanity.com/paper/2605.29400/) (2026)
- [Customer-Agent: Overcoming Context Limitations in Ultra-Long Shopping Trajectories via Tool-Augmented Agents and RLVR](https://mlsanity.com/paper/2606.07995/) (2026)
- [World Model-Guided Reinforcement Learning via Counterfactual User Engagement Simulation](https://mlsanity.com/paper/2609.01067/) (2026)
- [Scored vs. Generated Readouts in Behavioral Language Models: An Empirical Study of Elicitation Format](https://mlsanity.com/paper/2609.09882/) (2026)
- [Automatic multimodal UX improvement recommendations from LLM agent user simulations](https://mlsanity.com/paper/2609.22971/) (2026)
- [SimTrace: Grounded Multimodal User Trajectories Generation for Online User Modeling](https://mlsanity.com/paper/2609.38397/) (2026)
- [D-DOIT: Training-free Adaptation of Discrete Diffusion via Doob's h-Transform](https://mlsanity.com/paper/2610.04938/) (2026)
- [Socio-Foundation: A Model for Generalizable Individual Behavior Simulation via Hierarchical Capability Distillation](https://mlsanity.com/paper/2610.08967/) (2026)
- [Vela: User Behavior as Supervision Sparks LLM Intelligence](https://mlsanity.com/paper/2610.14709/) (2026)

## Abstract (arXiv)

Simulating step-wise human behavior with Large Language Models (LLMs) has become an emerging research direction, enabling applications in various practical domains. While prior methods, including prompting, supervised fine-tuning (SFT), and reinforcement learning (RL), have shown promise in modeling step-wise behavior, they primarily learn a population-level policy without conditioning on a user's persona, yielding generic rather than personalized simulations. In this work, we pose a critical question: how can LLM agents better simulate personalized user behavior? We introduce Customer-R1, an RL-based method for personalized, step-wise user behavior simulation in online shopping environments. Our policy is conditioned on an explicit persona, and we optimize next-step rationale and action generation via action correctness reward signals. Experiments on the OPeRA dataset emonstrate that Customer-R1 not only significantly outperforms prompting and SFT-based baselines in next-action prediction tasks, but also better matches users' action distribution, indicating higher fidelity in personalized behavior simulation.