# SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial Search

arXiv: https://arxiv.org/abs/2609.29803 · HTML: https://mlsanity.com/paper/2609.29803/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 25 of 25)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [SkillOpt: Executive Strategy for Self-Evolving Agent Skills](https://mlsanity.com/paper/2605.23904/) (2026)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [FIRST: Faster Improved Listwise Reranking with Single Token Decoding](https://mlsanity.com/paper/2406.15657/) (2024)
- [UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor](https://mlsanity.com/paper/2406.06519/) (2024)
- [SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision](https://mlsanity.com/paper/2606.01139/) (2026)
- [VLM as Policy: Common-Law Content Moderation Framework for Short Video Platform](https://mlsanity.com/paper/2504.14904/) (2025)
- [RankLLM: A Python Package for Reranking with LLMs](https://mlsanity.com/paper/2505.19284/) (2025)
- [A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment](https://mlsanity.com/paper/2504.12408/) (2025)
- [Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptation](https://mlsanity.com/paper/2310.02842/) (2023)
- [Optimizing Generative Ranking Relevance via Reinforcement Learning in Xiaohongshu Search](https://mlsanity.com/paper/2512.00968/) (2025)
- [Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills](https://mlsanity.com/paper/2607.22529/) (2026)
- [LREF: A Novel LLM-based Relevance Framework for E-commerce](https://mlsanity.com/paper/2503.09223/) (2025)
- [Batched Self-Consistency Improves LLM Relevance Assessment and Ranking](https://mlsanity.com/paper/2505.12570/) (2025)
- [BloomIntent: Automating Search Evaluation with LLM-Generated Fine-Grained User Intents](https://mlsanity.com/paper/2509.18641/) (2025)
- [TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance](https://mlsanity.com/paper/2510.08048/) (2025)
- [DARA: Few-shot Budget Allocation in Online Advertising via In-Context Decision Making with RL-Finetuned LLMs](https://mlsanity.com/paper/2601.14711/) (2026)
- [LLM-Driven Usefulness Judgment for Web Search Evaluation](https://mlsanity.com/paper/2504.14401/) (2025)
- [Redefining Retrieval Evaluation in the Era of LLMs](https://mlsanity.com/paper/2510.21440/) (2025)
- [LLM-Driven Usefulness Labeling for IR Evaluation](https://mlsanity.com/paper/2503.08965/) (2025)
- [An Efficient Framework for Whole-Page Reranking via Single-Modal Supervision](https://mlsanity.com/paper/2510.16803/) (2025)
- [As It Was: Aligning LLM Search Evaluation with Historical User Preferences](https://mlsanity.com/paper/2607.01040/) (2026)

## Built on by (most cited first, 1 of 1)

- [From Existing Methods to New Research Ideas: Two Perspectives on Recursive Self-Improvement](https://mlsanity.com/paper/2610.14625/) (2026)

## Abstract (arXiv)

Search quality evaluation provides essential supervision and diagnostic signals for the development and iteration of industrial search systems. Although large language models (LLMs) offer a scalable alternative to manual assessment, reliable automatic evaluation remains challenging: users experience search results at the page level, while the applicable evaluation criteria are multi-dimensional and continuously evolving. Packing all evaluation criteria into a unified prompt introduces irrelevant context and potential criterion interference, whereas internalizing them through post-training tightly couples rule updates with costly model retraining cycles. To address these issues, we propose Skill-routed Evaluation with Evolvable Knowledge (SEEK). Specifically, SEEK externalizes specific search evaluation criteria into a skill bank, dynamically routes relevant skills for each query-result list pair, and employs a task-adapted listwise evaluator to produce page-level judgments and failure mode attribution. A two-stage training pipeline teaches the evaluator to align evaluation criteria with human preferences, while a replay-gated skill bank allows recurring evaluation knowledge gaps to be incorporated without model retraining. Experiments on industrial short-video search show that SEEK improves listwise quality evaluation accuracy and achieves significant progress in attribution diagnosis. SEEK has been deployed at Kuaishou, a short-video platform with over 400 million daily active users, significantly improving the scale and quality of online search evaluation.