# From Clicks to Preference: A Multi-stage Alignment Framework for Generative Query Suggestion in Conversational System

arXiv: https://arxiv.org/abs/2508.15811 · HTML: https://mlsanity.com/paper/2508.15811/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 15 Aug 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [Secrets of RLHF in Large Language Models Part II: Reward Modeling](https://mlsanity.com/paper/2401.06080/) (2024)
- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [Checklists Are Better Than Reward Models For Aligning Language Models](https://mlsanity.com/paper/2507.18624/) (2025)
- [Large Language Models as Zero-Shot Conversational Recommenders](https://mlsanity.com/paper/2308.10053/) (2023)
- [OffsetBias: Leveraging Debiased Data for Tuning Evaluators](https://mlsanity.com/paper/2407.06551/) (2024)
- [A Hierarchical Recurrent Encoder-Decoder For Generative Context-Aware Query Suggestion](https://mlsanity.com/paper/1507.02221/) (2015)
- [Evaluation of Session-based Recommendation Algorithms](https://mlsanity.com/paper/1803.09587/) (2018)
- [Learning to Attend, Copy, and Generate for Session-Based Query Suggestion](https://mlsanity.com/paper/1708.03418/) (2017)
- [Empirical Analysis of Session-Based Recommendation Algorithms](https://mlsanity.com/paper/1910.12781/) (2019)
- [Zero-shot Clarifying Question Generation for Conversational Search](https://mlsanity.com/paper/2301.12660/) (2023)
- [Multimodal Query Suggestion with Multi-Agent Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2402.04867/) (2024)
- [CTR-Guided Generative Query Suggestion in Conversational Search](https://mlsanity.com/paper/2507.04072/) (2025)
- [From Prompting to Alignment: A Generative Framework for Query Recommendation](https://mlsanity.com/paper/2504.10208/) (2025)

## Built on by (most cited first, 9 of 9)

- [What to Edit Next: Visually Aligned Image-Editing Follow-Up Suggestions in Conversational Systems](https://mlsanity.com/paper/2608.07565/) (2026)
- [Improve Large Language Model Systems with User Logs](https://mlsanity.com/paper/2602.06470/) (2026)
- [IceBreaker for Conversational Agents: Breaking the First-Message Barrier with Personalized Starters](https://mlsanity.com/paper/2604.18375/) (2026)
- [Don't Let Bandit Feedback Pull Continual LLM-Recommender Updates Off Target](https://mlsanity.com/paper/2605.18899/) (2026)
- [OnePred: Next-Query Prediction via Recursive Intent Memory in Multi-Turn Conversations](https://mlsanity.com/paper/2605.23668/) (2026)
- [Generative Query Suggestion via Intent Coverage and Query-Level Credit Assignment](https://mlsanity.com/paper/2609.19209/) (2026)
- [Bridging Passive and Active: Enhancing Conversation Starter Recommendation via Active Expression Modeling](https://mlsanity.com/paper/2605.05855/) (2026)
- [GenMatch: An End-to-End Generative Matching Framework for Micro-View Order-Dispatching in Ride-Hailing](https://mlsanity.com/paper/2608.19751/) (2026)
- [Search-Aware Reinforcement Learning for Multi-Component Query Understanding in Roblox Game Search](https://mlsanity.com/paper/2609.30177/) (2026)

## Abstract (arXiv)

Generative query suggestion using large language models offers a powerful way to enhance conversational systems, but aligning outputs with nuanced user preferences remains a critical challenge. To address this, we introduce a multi-stage framework designed for progressive alignment between the generation policy and user intent. Our pipeline begins with prompt engineering as a cold-start strategy, followed by the Supervised Fine-Tuning stage, in which we introduce a distillation method on click logs to create a robust foundational model. To better model user preferences while capturing their inherent uncertainty, we develop a Gaussian Reward Model (GaRM) that represents user preferences as probability distributions rather than point estimates. Finally, we employ reinforcement learning to align the generation policy with these preferences, guided by a composite reward function that integrates GaRM with auxiliary heuristics to mitigate reward hacking. To maintain training stability, this process is enhanced by a novel out-of-distribution regularization method and a two-stage reward fusion technique. Extensive experiments demonstrate that our framework significantly outperforms baselines on both automatic and human evaluations and yields a 34% relative increase in user engagement as measured by click-through rate in live A/B tests.