mlsanityData to 7 Oct 2026

Ask Optimal Questions: Aligning Large Language Models with Retriever's Preference in Conversation

arXiv 2402.11827 ↗First posted 19 Feb 2024cs.IR cs.CLcited by 23

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Conversational search, unlike single-turn retrieval tasks, requires understanding the current question within a dialogue context. The common approach of rewrite-then-retrieve aims to decontextualize questions to be self-sufficient for off-the-shelf retrievers, but most existing methods produce sub-optimal query rewrites due to the limited ability to incorporate signals from the retrieval results. To overcome this limitation, we present a novel framework RetPO (Retriever's Preference Optimization), which is designed to optimize a language model (LM) for reformulating search queries in line with the preferences of the target retrieval systems. The process begins by prompting a large LM to produce various potential rewrites and then collects retrieval performance for these rewrites as the retrievers' preferences. Through the process, we construct a large-scale dataset called RF collection, containing Retrievers' Feedback on over 410K query rewrites across 12K conversations. Furthermore, we fine-tune a smaller LM on this dataset to align it with the retrievers' feedback. Our resulting model demonstrates superiority on two benchmarks, surpassing the previous state-of-the-art performance of rewrite-then-retrieve approaches.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 61 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

36 of 61 references.

References · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  12. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
24 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  2. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  3. 2018QuAC : Question Answering in Contextcited by 379
  4. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  5. 2019ScispaCy: Fast and Robust Models for Biomedical Natural Language Processingcited by 244
  6. 2023Fine-tuning Language Models for Factualitycited by 219
  7. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  8. 2020Open-Domain Question Answering Goes Conversational via Question Rewritingcited by 147
  9. 2020Open-Retrieval Conversational Question Answeringcited by 99
  10. 2021TopiOCQA: Open-domain Conversational Question Answering with Topic Switchingcited by 95
  11. 2020Question Rewriting for Conversational Question Answeringcited by 93
  12. 2020Few-Shot Generative Conversational Query Rewritingcited by 85
  13. 2023Aligning Large Language Models through Synthetic Feedbackcited by 75
  14. 2023Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Searchcited by 74
  15. 2021Few-Shot Conversational Dense Retrievalcited by 71
  16. 2023Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewritingcited by 69
  17. 2021CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learningcited by 68
  18. 2023ConvGQR: Generative Query Reformulation for Conversational Searchcited by 59
  19. 2020Query Resolution for Conversational Search with Limited Supervisioncited by 52
  20. 2018Pytrec_eval: An Extremely Fast Python Interface to trec_evalcited by 46
  21. 2020Conversational Question Reformulation via Sequence-to-Sequence Architectures and Pretrained Language Modelscited by 45
  22. 2021Contextualized Query Embeddings for Conversational Searchcited by 36
  23. 2022Saving Dense Retriever from Shortcut Dependency in Conversational Searchcited by 18
  24. 2022Generating Information-Seeking Conversations from Unlabeled Documentscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 9; 2026: 5.

23 papers.

Sort:

By area:

  1. 2025Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learningcited by 32
  2. 2024CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Searchcited by 29
  3. 2024A Survey of Conversational Searchcited by 22
  4. 2024ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrievalcited by 18
  5. 2024Adaptive Query Rewriting: Aligning Rewriters through Marginal Probability of Conversational Answerscited by 15
  6. 2024AdaCQR: Enhancing Query Reformulation for Conversational Search via Sparse and Dense Retrieval Alignmentcited by 14
  7. 2025Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applicationscited by 13
  8. 2025UniConv: Unifying Retrieval and Response Generation for Large Language Models in Conversationscited by 11
  9. 2024Crafting the Path: Robust Query Rewriting for Information Retrievalcited by 4
  10. 2024Broadening Access to Simulations for End-Users via Large Language Models: Challenges and Opportunitiescited by 4
  11. 2025ConvMix: A Mixed-Criteria Data Augmentation Framework for Conversational Dense Retrievalcited by 4
  12. 2025ICR: Iterative Clarification and Rewriting for Conversational Searchcited by 4
11 more
  1. 2024Zero-Shot Dense Retrieval with Embeddings from Relevance Feedbackcited by 3
  2. 2025AdaRewriter: Unleashing the Power of Prompting-based Conversational Query Reformulation via Test-Time Adaptationcited by 2
  3. 2024Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generationcited by 1
  4. 2026Multi-Faceted Self-Consistent Preference Alignment for Query Rewriting in Conversational Searchcited by 1
  5. 2025References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulationcited by 0
  6. 2025Building Open-Retrieval Conversational Question Answering Systems by Generating Synthetic Data and Decontextualizing User Questionscited by 0
  7. 2025Adaptive Personalized Conversational Information Retrievalcited by 0
  8. 2026When should I search more: Adaptive Complex Query Optimization with Reinforcement Learningcited by 0
  9. 2026Factorized Hypothesis Search for Evidence-to-Taxonomy Retrievalcited by 0
  10. 2026Don't Scroll Back: Missing-Evidence Memory for Streaming Dialogue Summarizationcited by 0
  11. 2026CompCQR: Compositional Query Generation for Training-Free Conversational Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RAC: Retrieval-Augmented Clarification for Faithful Conversational Search
  2. 2026MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks
  3. 2024Rewriting Conversational Utterances with Instructed Large Language Models
  4. 2023Query Rewriting for Retrieval-Augmented Large Language Models
  5. 2024Learning When to Retrieve, What to Rewrite, and How to Respond in Conversational QA
  6. 2023CONVERSER: Few-Shot Conversational Dense Retrieval with Synthetic Data Generation
  7. 2025ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning
  8. 2026uva-irlab-conv at SemEval-2026 Task 8: Multi-Turn RAG with Learned Sparse Retrieval and Listwise Reranking