# TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance

arXiv: https://arxiv.org/abs/2510.08048 · HTML: https://mlsanity.com/paper/2510.08048/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)
- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs](https://mlsanity.com/paper/2406.18629/) (2024)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [Teaching Large Language Models to Reason with Reinforcement Learning](https://mlsanity.com/paper/2403.04642/) (2024)
- [Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library](https://mlsanity.com/paper/2506.06122/) (2025)
- [LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning](https://mlsanity.com/paper/2410.02884/) (2024)
- [On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting](https://mlsanity.com/paper/2508.11408/) (2025)
- [Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback](https://mlsanity.com/paper/2506.03106/) (2025)
- [GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models](https://mlsanity.com/paper/2504.09696/) (2025)
- [Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge](https://mlsanity.com/paper/2501.18099/) (2025)
- [StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason](https://mlsanity.com/paper/2507.02841/) (2025)
- [NGRPO: Negative-enhanced Group Relative Policy Optimization](https://mlsanity.com/paper/2509.18851/) (2025)
- [Exploiting Tree Structure for Credit Assignment in RL Training of LLMs](https://mlsanity.com/paper/2509.18314/) (2025)
- [Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models](https://mlsanity.com/paper/2506.13923/) (2025)
- [LREF: A Novel LLM-based Relevance Framework for E-commerce](https://mlsanity.com/paper/2503.09223/) (2025)
- [LLM-Driven Intrinsic Motivation for Sparse Reward Reinforcement Learning](https://mlsanity.com/paper/2508.18420/) (2025)

## Built on by (most cited first, 4 of 4)

- [Synthetic Data Powers Product Retrieval for Long-tail Knowledge-Intensive Queries in E-commerce Search](https://mlsanity.com/paper/2602.23620/) (2026)
- [K-CARE: Knowledge-driven Symmetrical Contextual Anchoring and Analogical Prototype Reasoning for E-commerce Relevance](https://mlsanity.com/paper/2604.25683/) (2026)
- [SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial Search](https://mlsanity.com/paper/2609.29803/) (2026)
- [From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction](https://mlsanity.com/paper/2610.16569/) (2026)

## Abstract (arXiv)

Query-product relevance prediction is fundamental to e-commerce search and has become even more critical in the era of AI-powered shopping, where semantic understanding and complex reasoning directly shape the user experience and business conversion. Large Language Models (LLMs) enable generative, reasoning-based approaches, typically aligned via supervised fine-tuning (SFT) or preference optimization methods like Direct Preference Optimization (DPO). However, the increasing complexity of business rules and user queries exposes the inability of existing methods to endow models with robust reasoning capacity for long-tail and challenging cases. Efforts to address this via reinforcement learning strategies like Group Relative Policy Optimization (GRPO) often suffer from sparse terminal rewards, offering insufficient guidance for multi-step reasoning and slowing convergence. To address these challenges, we propose TaoSR-AGRL, an Adaptive Guided Reinforcement Learning framework for LLM-based relevance prediction in Taobao Search Relevance. TaoSR-AGRL introduces two key innovations: (1) Rule-aware Reward Shaping, which decomposes the final relevance judgment into dense, structured rewards aligned with domain-specific relevance criteria; and (2) Adaptive Guided Replay, which identifies low-accuracy rollouts during training and injects targeted ground-truth guidance to steer the policy away from stagnant, rule-violating reasoning patterns toward compliant trajectories. TaoSR-AGRL was evaluated on large-scale real-world datasets and through online side-by-side human evaluations on Taobao Search. It consistently outperforms DPO and standard GRPO baselines in offline experiments, improving relevance accuracy, rule adherence, and training stability. The model trained with TaoSR-AGRL has been successfully deployed in the main search scenario on Taobao, serving hundreds of millions of users.