# Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application

arXiv: https://arxiv.org/abs/1803.00710 · HTML: https://mlsanity.com/paper/1803.00710/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2018.

## Built on by (most cited first, 30 of 34)

- [Advances and Challenges in Conversational Recommender Systems: A Survey](https://mlsanity.com/paper/2101.09459/) (2021)
- [Contrastive language and vision learning of general fashion concepts](https://mlsanity.com/paper/2204.03972/) (2022)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [Fair ranking: a critical review, challenges, and future directions](https://mlsanity.com/paper/2201.12662/) (2022)
- [Choosing the Best of Both Worlds: Diverse and Novel Recommendations through Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2110.15097/) (2021)
- [Supervised Advantage Actor-Critic for Recommender Systems](https://mlsanity.com/paper/2111.03474/) (2021)
- [Learning to Ask: Conversational Product Search via Representation Learning](https://mlsanity.com/paper/2411.14466/) (2024)
- [Adversarial Mixture Of Experts with Category Hierarchy Soft Constraint](https://mlsanity.com/paper/2007.12349/) (2020)
- [Should I send this notification? Optimizing push notifications decision making by modeling the future](https://mlsanity.com/paper/2202.08812/) (2022)
- [RL4RS: A Real-World Dataset for Reinforcement Learning based Recommender System](https://mlsanity.com/paper/2110.11073/) (2021)
- [Reinforcement Learning with a Disentangled Universal Value Function for Item Recommendation](https://mlsanity.com/paper/2104.02981/) (2021)
- [Understanding User Intent Modeling for Conversational Recommender Systems: A Systematic Literature Review](https://mlsanity.com/paper/2308.08496/) (2023)
- [On the Effectiveness of Unlearning in Session-Based Recommendation](https://mlsanity.com/paper/2312.14447/) (2023)
- [BCRLSP: An Offline Reinforcement Learning Framework for Sequential Targeted Promotion](https://mlsanity.com/paper/2207.07790/) (2022)
- [Interpretable performance analysis towards offline reinforcement learning: A dataset perspective](https://mlsanity.com/paper/2105.05473/) (2021)
- [UOEP: User-Oriented Exploration Policy for Enhancing Long-Term User Experiences in Recommender Systems](https://mlsanity.com/paper/2401.09034/) (2024)
- [Sequence Adaptation via Reinforcement Learning in Recommender Systems](https://mlsanity.com/paper/2108.01442/) (2021)
- [Conditional Sequential Slate Optimization](https://mlsanity.com/paper/2108.05618/) (2021)
- [Top-K Ranking Deep Contextual Bandits for Information Selection Systems](https://mlsanity.com/paper/2201.13287/) (2022)
- [Sequential Search with Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2202.00245/) (2022)
- [Composing Efficient, Robust Tests for Policy Selection](https://mlsanity.com/paper/2306.07372/) (2023)
- [Deep Reinforcement Learning for Ranking Utility Tuning in the Ad Recommender System at Pinterest](https://mlsanity.com/paper/2509.05292/) (2025)
- [Energy-Guided Diffusion Sampling for Long-Term User Behavior Prediction in Reinforcement Learning-based Recommendation](https://mlsanity.com/paper/2510.12815/) (2025)
- [Deep Reinforcement Learning-Based Product Recommender for Online Advertising](https://mlsanity.com/paper/2102.00333/) (2021)
- [D2RLIR : an improved and diversified ranking function in interactive recommendation systems based on deep reinforcement learning](https://mlsanity.com/paper/2110.15089/) (2021)
- [Who Are the Best Adopters? User Selection Model for Free Trial Item Promotion](https://mlsanity.com/paper/2202.09508/) (2022)
- [Towards Validating Long-Term User Feedbacks in Interactive Recommendation Systems](https://mlsanity.com/paper/2308.11137/) (2023)
- [Ranking Policy Learning via Marketplace Expected Value Estimation From Observational Data](https://mlsanity.com/paper/2410.04568/) (2024)
- [Extracting Important Tokens in E-Commerce Queries with a Tag Interaction-Aware Transformer Model](https://mlsanity.com/paper/2507.10385/) (2025)
- [DiffusionGS: Generative Search with Query Conditioned Diffusion in Kuaishou](https://mlsanity.com/paper/2508.17754/) (2025)

## Abstract (arXiv)

In e-commerce platforms such as Amazon and TaoBao, ranking items in a search session is a typical multi-step decision-making problem. Learning to rank (LTR) methods have been widely applied to ranking problems. However, such methods often consider different ranking steps in a session to be independent, which conversely may be highly correlated to each other. For better utilizing the correlation between different ranking steps, in this paper, we propose to use reinforcement learning (RL) to learn an optimal ranking policy which maximizes the expected accumulative rewards in a search session. Firstly, we formally define the concept of search session Markov decision process (SSMDP) to formulate the multi-step ranking problem. Secondly, we analyze the property of SSMDP and theoretically prove the necessity of maximizing accumulative rewards. Lastly, we propose a novel policy gradient algorithm for learning an optimal ranking policy, which is able to deal with the problem of high reward variance and unbalanced reward distribution of an SSMDP. Experiments are conducted in simulation and TaoBao search engine. The results demonstrate that our algorithm performs much better than online LTR methods, with more than 40% and 30% growth of total transaction amount in the simulation and the real application, respectively.