# RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce

arXiv: https://arxiv.org/abs/2609.38974 · HTML: https://mlsanity.com/paper/2609.38974/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agentic recommender systems, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Placed in

- LLM agents and tool use / [Agentic recommender systems](https://mlsanity.com/agents/agentic-recommender-systems/)

## Builds on (most cited first, 22 of 22)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems](https://mlsanity.com/paper/2310.09233/) (2023)
- [DeepShop: A Benchmark for Deep Research Shopping Agents](https://mlsanity.com/paper/2506.02839/) (2025)
- [PersonaX: A Recommendation Agent Oriented User Modeling Framework for Long Behavior Sequence](https://mlsanity.com/paper/2503.02398/) (2025)
- [OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation](https://mlsanity.com/paper/2506.05606/) (2025)
- [Towards Next-Generation Recommender Systems: A Benchmark for Personalized Recommendation Assistant with LLMs](https://mlsanity.com/paper/2503.09382/) (2025)
- [Reason-to-Recommend: Using Interaction-of-Thought Reasoning to Enhance LLM Recommendation](https://mlsanity.com/paper/2506.05069/) (2025)
- [RecoWorld: Building Simulated Environments for Agentic Recommender Systems](https://mlsanity.com/paper/2509.10397/) (2025)
- [Agentic Feedback Loop Modeling Improves Recommendation and User Simulation](https://mlsanity.com/paper/2410.20027/) (2024)
- [Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors](https://mlsanity.com/paper/2404.03304/) (2024)
- [RecThinker: An Agentic Framework for Tool-Augmented Reasoning in Recommendation](https://mlsanity.com/paper/2603.09843/) (2026)
- [SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use](https://mlsanity.com/paper/2607.01874/) (2026)
- [VCSearch: Bridging the Gap Between Well-Defined and Ill-Defined Problems in Mathematical Reasoning](https://mlsanity.com/paper/2406.05055/) (2024)
- [SAPIENT: Mastering Multi-turn Conversational Recommendation with Strategic Planning and Monte Carlo Tree Search](https://mlsanity.com/paper/2410.09580/) (2024)
- [Towards Personalized Conversational Sales Agents: Contextual User Profiling for Strategic Action](https://mlsanity.com/paper/2504.08754/) (2025)
- [Search-Based Interaction For Conversation Recommendation via Generative Reward Model Based Simulated User](https://mlsanity.com/paper/2504.20458/) (2025)
- [Self-Evolving Neuro-Symbolic Skills for Tool-Augmented Spatial Reasoning](https://mlsanity.com/paper/2608.07955/) (2026)
- [EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commerce](https://mlsanity.com/paper/2601.02752/) (2026)
- [Unleashing the Native Recommendation Potential: LLM-Based Generative Recommendation via Structured Term Identifiers](https://mlsanity.com/paper/2601.06798/) (2026)
- [A Framework for Generating Conversational Recommendation Datasets from Behavioral Interactions](https://mlsanity.com/paper/2506.17285/) (2025)
- [ComboShoppingBench: Evaluating LLM Agents for Budget-Constrained Basket Shopping with Coupons](https://mlsanity.com/paper/2608.09282/) (2026)
- [Fashion-AlterEval: A Dataset for Improved Evaluation of Conversational Recommendation Systems with Alternative Relevant Items](https://mlsanity.com/paper/2507.18017/) (2025)

## Abstract (arXiv)

Large language models are reshaping ecommerce from static recommenders into interactive shopping assistants, yet real-world shopping requires session-level decision support: users reveal and revise constraints, coordinate multiple goals, and expect product-grounded recommendations over a full conversation. Existing benchmarks are mostly outcome-oriented or execution-oriented, leaving this evolving decision process under-evaluated. We introduce REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation. Our analysis shows that current systems produce locally plausible responses but struggle with state tracking, constraint updating, and grounded convergence, especially under ambiguous intent, bundle, and multi-intent scenarios. We further propose REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, catalog-grounded retrieval, and runtime guards. Experiments show that REALSHOP_AGENT consistently outperforms strong baselines on REALWORLDSHOP.