mlsanityData to 14 Oct 2026

Aligning Large Language Models with Searcher Preferences

arXiv 2603.10473 ↗First posted 11 Mar 2026cs.CL cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

The paradigm shift from item-centric ranking to answer-centric synthesis is redefining the role of search engines. While recent industrial progress has applied generative techniques to closed-set item ranking in e-commerce, research and deployment of open-ended generative search on large content platforms remain limited. This setting introduces challenges, including robustness to noisy retrieval, non-negotiable safety guarantees, and alignment with diverse user needs. In this work, we introduce SearchLLM, the first large language model (LLM) for open-ended generative search. We design a hierarchical, multi-dimensional reward system that separates bottom-line constraints, including factual grounding, basic answer quality and format compliance, from behavior optimization objectives that promote robustness to noisy retrieval and alignment with user needs. Concretely, our reward model evaluates responses conditioned on the user query, session history, and retrieved evidence set, combining rule-based checks with human-calibrated LLM judges to produce an interpretable score vector over these dimensions. We introduce a Gated Aggregation Strategy to derive the training reward for optimizing SearchLLM with Group Relative Policy Optimization (GRPO). We deploy SearchLLM in the AI search entry of RedNote. Offline evaluations and online A/B tests show improved generation quality and user engagement, increasing Valid Consumption Rate by 1.03% and reducing Re-search Rate by 2.81%, while upholding strict safety and reliability standards.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 49 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

32 of 49 references.

References · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  9. 2019Passage Re-ranking with BERTcited by 645
  10. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  12. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
20 more
  1. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  2. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  3. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  4. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
  5. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  6. 2023Query2doc: Query Expansion with Large Language Modelscited by 149
  7. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  8. 2024Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphscited by 78
  9. 2023Large Language Model based Long-tail Query Rewriting in Taobao Searchcited by 62
  10. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  11. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
  12. 2023Scalable and Effective Generative Information Retrievalcited by 39
  13. 2025OneSearch: A Preliminary Exploration of the Unified End-to-End Generative Framework for E-commerce Searchcited by 25
  14. 2024The Use of Generative Search Engines for Knowledge Work and Complex Taskscited by 18
  15. 2025Generative Retrieval and Alignment Model: A New Paradigm for E-commerce Retrievalcited by 13
  16. 2024AFDGCF: Adaptive Feature De-correlation Graph Collaborative Filtering for Recommendationscited by 12
  17. 2025Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Schedulingcited by 10
  18. 2024Structure-Enhanced Protein Instruction Tuning: Towards General-Purpose Protein Understanding with LLMscited by 4
  19. 2025Towards Context-aware Reasoning-enhanced Generative Searching in E-commercecited by 3
  20. 2026Anti-Length Shift: Dynamic Outlier Truncation for Training Efficient Reasoning Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utilitycited by 4
  2. 2026ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PageLLM: A Multi-Grained Reward Framework for Whole-Page Optimization with Large Language Models
  2. 2023Large Search Model: Redefining Search Stack in the Era of LLMs
  3. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
  4. 2025From Clicks to Preference: A Multi-stage Alignment Framework for Generative Query Suggestion in Conversational System
  5. 2024Generative Retrieval with Preference Optimization for E-commerce Search
  6. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searching
  7. 2026Tool-Adaptive LLM Reranker
  8. 2025Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search