References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulation

arXiv 2505.06552 ↗First posted 10 May 2025cs.CL cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Conversational query reformulation (CQR) has become indispensable for improving retrieval in dialogue-based applications. However, existing approaches typically rely on reference passages for optimization, which are impractical to acquire in real-world scenarios. To address this limitation, we introduce a novel reference-free preference optimization framework DualReform that generates pseudo reference passages from commonly-encountered conversational datasets containing only queries and responses. DualReform attains this goal through two key innovations: (1) response-based inference, where responses serve as proxies to infer pseudo reference passages, and (2) response refinement via the dual-role of CQR, where a CQR model refines responses based on the shared objectives between response refinement and CQR. Despite not relying on reference passages, DualReform achieves 96.9--99.1% of the retrieval accuracy attainable only with reference passages and surpasses the state-of-the-art method by up to 31.6%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 58 references that are in the map. First posted on arXiv 10 May 2025.

Builds on

47 of 58 references.

Closest ancestors · 1

Same direction, newest first

2024Ask Optimal Questions: Aligning Large Language Models with Retriever's Preference in Conversation19 Feb

Other references · 46

2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2017Proximal Policy Optimization Algorithmscited by 14,441
2024The Llama 3 Herd of Modelscited by 14,192
2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
2017Billion-scale similarity search with GPUscited by 2,477
2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304

34 more

2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
2024Self-Rewarding Language Modelscited by 654
2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
2021Large Dual Encoders Are Generalizable Retrieverscited by 454
2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
2020Open-Domain Question Answering Goes Conversational via Question Rewritingcited by 147
2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
2020Open-Retrieval Conversational Question Answeringcited by 99
2021TopiOCQA: Open-domain Conversational Question Answering with Topic Switchingcited by 95
2020Question Rewriting for Conversational Question Answeringcited by 93
2022Self-Training: A Surveycited by 85
2023Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Searchcited by 74
2021Few-Shot Conversational Dense Retrievalcited by 71
2023Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewritingcited by 69
2021CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learningcited by 68
2023ConvGQR: Generative Query Reformulation for Conversational Searchcited by 59
2021A Theory of Label Propagation for Subpopulation Shiftcited by 48
2018Pytrec_eval: An Extremely Fast Python Interface to trec_evalcited by 46
2020Conversational Question Reformulation via Sequence-to-Sequence Architectures and Pretrained Language Modelscited by 45
2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 38
2021Contextualized Query Embeddings for Conversational Searchcited by 36
2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31
2024Retrieval-augmented generation in multilingual settingscited by 27
2024Learning to Summarize from LLM-generated Feedbackcited by 24
2024A Survey on Human Preference Learning for Large Language Modelscited by 23
2023Knowledge-Augmented Language Model Verificationcited by 21
2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modelingcited by 21
2023Robust Data Pruning under Label Noise via Maximizing Re-labeling Accuracycited by 20
2024Geometric-Averaged Preference Optimization for Soft Preference Labelscited by 18
2024AdaCQR: Enhancing Query Reformulation for Conversational Search via Sparse and Dense Retrieval Alignmentcited by 14
2024RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answeringcited by 13
2023Enhancing Abstractiveness of Summarization Models through Calibrated Distillationcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way