mlsanityData to 7 Oct 2026

Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning

arXiv 1805.10000 ↗First posted 25 May 2018cs.AIcited by 80

Not on the map. Why?

Cited by 23 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Applying reinforcement learning in physical-world tasks is extremely challenging. It is commonly infeasible to sample a large number of trials, as required by current reinforcement learning methods, in a physical environment. This paper reports our project on using reinforcement learning for better commodity search in Taobao, one of the largest online retail platforms and meanwhile a physical environment with a high sampling cost. Instead of training reinforcement learning in Taobao directly, we present our approach: first we build Virtual Taobao, a simulator learned from historical customer behavior data through the proposed GAN-SD (GAN for Simulating Distributions) and MAIL (multi-agent adversarial imitation learning), and then we train policies in Virtual Taobao with no physical costs in which ANC (Action Norm Constraint) strategy is proposed to reduce over-fitting. In experiments, Virtual Taobao is trained from hundreds of millions of customers' records, and its properties are compared with the real environment. The results disclose that Virtual Taobao faithfully recovers important properties of the real environment. We also show that the policies trained in Virtual Taobao can have significantly superior online performance to the traditional supervised approaches. We hope our work could shed some light on reinforcement learning applications in complex physical environments.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 12; 2022: 18; 2023: 20; 2024: 13; 2025: 11; 2026: 5.

80 papers.

Sort:

By area:

  1. 2023User Behavior Simulation with Large Language Model based Agentscited by 136
  2. 2023On Generative Agents in Recommendationcited by 88
  3. 2022A Survey on Model-based Reinforcement Learningcited by 65
  4. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  5. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 47
  6. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 45
  7. 2023A Survey on Causal Reinforcement Learningcited by 43
  8. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 36
  9. 2022Estimating and Penalizing Induced Preference Shifts in Recommender Systemscited by 29
  10. 2021RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystemscited by 21
  11. 2022Fair ranking: a critical review, challenges, and future directionscited by 21
  12. 2022PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagementcited by 21
38 more
  1. 2023Learning to Simulate Daily Activities via Modeling Dynamic Human Needscited by 20
  2. 2021A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directionscited by 19
  3. 2022Offline Reinforcement Learning with Causal Structured World Modelscited by 19
  4. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitionscited by 16
  5. 2021Derivative-Free Reinforcement Learning: A Reviewcited by 15
  6. 2022Rethinking ValueDice: Does It Really Improve Performance?cited by 15
  7. 2021Deep Exploration for Recommendation Systemscited by 14
  8. 2022Dynamic Regret of Online Markov Decision Processescited by 13
  9. 2022Adversarial Counterfactual Environment Model Learningcited by 12
  10. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learningcited by 12
  11. 2023Causal Decision Transformer for Recommender Systems via Offline Reinforcement Learningcited by 11
  12. 2024Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspectivecited by 11
  13. 2024SUBER: An RL Environment with Simulated Human Behavior for Recommender Systemscited by 11
  14. 2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systemscited by 10
  15. 2021Supervised Advantage Actor-Critic for Recommender Systemscited by 8
  16. 2022Offline Reinforcement Learning for Mobile Notificationscited by 8
  17. 2023How To Guide Your Learner: Imitation Learning with Active Adaptive Expert Involvementcited by 8
  18. 2024EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systemscited by 8
  19. 2022RITA: Boost Driving Simulators with Realistic Interactive Traffic Flowcited by 7
  20. 2019Imitation Learning from Pixel-Level Demonstrations by HashRewardcited by 6
  21. 2021Generative Adversarial Reward Learning for Generalized Behavior Tendency Inferencecited by 6
  22. 2022Model-based Reinforcement Learning with Multi-step Plan Value Estimationcited by 6
  23. 2024Provably and Practically Efficient Adversarial Imitation Learning with General Function Approximationcited by 6
  24. 2026Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Tracescited by 6
  25. 2021We Know What You Want: An Advertising Strategy Recommender System for Online Advertisingcited by 5
  26. 2021RL4RS: A Real-World Dataset for Reinforcement Learning based Recommender Systemcited by 5
  27. 2022Reinforcement Learning in Practice: Opportunities and Challengescited by 5
  28. 2022Synthetic Data-Based Simulators for Recommender Systems: A Surveycited by 5
  29. 2023Sim2Rec: A Simulator-based Decision-making Approach to Optimize Real-World Long-term User Engagement in Sequential Recommender Systemscited by 5
  30. 2023AURO: Reinforcement Learning for Adaptive User Retention Optimization in Recommender Systemscited by 5
  31. 2024Lusifer: LLM-based User SImulated Feedback Environment for online Recommender systemscited by 5
  32. 2021Reinforcement Learning with a Disentangled Universal Value Function for Item Recommendationcited by 4
  33. 2022Context Uncertainty in Contextual Bandits with Applications to Recommender Systemscited by 4
  34. 2023Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspectivecited by 4
  35. 2025Beyond Static Testbeds: An Interaction-Centric Agent Simulation Platform for Dynamic Recommender Systemscited by 4
  36. 2021Imitate TheWorld: A Search Engine Simulation Platformcited by 3
  37. 2022Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learningcited by 3
  38. 2023Marketing Budget Allocation with Offline Constrained Deep Reinforcement Learningcited by 3

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants
  2. 2025Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning
  3. 2025TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance
  4. 2026LEAPS: An LLM-Empowered Adaptive Plugin in Taobao AI Search ↗
  5. 2026Towards Faithful Simulation of Human Shopping Behavior ↗
  6. 2025EPR-GAIL: An EPR-Enhanced Hierarchical Imitation Learning Framework to Simulate Complex User Consumption Behaviors
  7. 2025ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents
  8. 2022Profitable Strategy Design by Using Deep Reinforcement Learning for Trades on Cryptocurrency Markets