Search-Aware Reinforcement Learning for Multi-Component Query Understanding in Roblox Game Search
On the map: Deep reinforcement learning methods → RL training of language and agent models
Abstract (arXiv)
Query understanding (QU) plays a critical role in production search systems, translating raw user queries into search execution plans that drive downstream retrieval and ranking. While large language models (LLMs) have enabled QU to be framed as a structured multi-task generation problem (e.g., intent classification, query expansion), optimizing such models to produce search-engine-coupled outputs remains challenging: static, label-based supervision fails to capture how each component actually interacts with the underlying search pipeline to affect downstream performance. We present a search-aware reinforcement learning (RL) framework for QU based on a distill-then-RL paradigm. Teacher-student supervised fine-tuning (SFT) first yields a well-formed, schema-compliant policy initialization. The RL stage then optimizes each QU component with rewards derived from live interaction with the search engine, tailored to that component's operational role, rather than a single reward tied to the final search outcome. Experiments on Roblox search show that this component-specific optimization improves both per-component utility and downstream search quality, raising NDCG@20 by 8.9 points over the SFT policy and by 3.5 points over training with a single end-to-end reward.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 27 references that are in the map. First posted on arXiv 24 Sep 2026.
Builds on
12 of 27 references.
References · 12
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
- 2016Sequence-Level Knowledge Distillationcited by 801
- 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
- 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 88
- 2025Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Rewardcited by 25
- 2023QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Searchcited by 10
- 2025From Clicks to Preference: A Multi-stage Alignment Framework for Generative Query Suggestion in Conversational Systemcited by 9
- 2026QP-OneModel: A Unified Generative LLM for Multi-Task Query Understanding in Xiaohongshu Searchcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
- 2025AI-SearchPlanner: Modular Agentic Search via Pareto-Optimal Multi-Objective Reinforcement Learning
- 2025AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learning
- 2025DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning
- 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents
- 2026CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search
- 2026PROGRESS: Coverage-guided RL to Train Search-augmented LLM Agent
- 2025DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi-Reward Reinforcement Learning