# MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language Models

arXiv: https://arxiv.org/abs/2408.17072 · HTML: https://mlsanity.com/paper/2408.17072/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 30 Aug 2024.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)
- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [Open-Domain Question Answering Goes Conversational via Question Rewriting](https://mlsanity.com/paper/2010.04898/) (2020)
- [Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPT](https://mlsanity.com/paper/2304.11116/) (2023)
- [Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting](https://mlsanity.com/paper/2310.09716/) (2023)
- [TREC CAsT 2019: The Conversational Assistance Track Overview](https://mlsanity.com/paper/2003.13624/) (2020)
- [ConvGQR: Generative Query Reformulation for Conversational Search](https://mlsanity.com/paper/2305.15645/) (2023)
- [Conversational Question Reformulation via Sequence-to-Sequence Architectures and Pretrained Language Models](https://mlsanity.com/paper/2004.01909/) (2020)
- [Boosting Conversational Question Answering with Fine-Grained Retrieval-Augmentation and Self-Check](https://mlsanity.com/paper/2403.18243/) (2024)
- [Retrieval Augmented Generation for Domain-specific Question Answering](https://mlsanity.com/paper/2404.14760/) (2024)
- [Retrieval-Augmented Generative Question Answering for Event Argument Extraction](https://mlsanity.com/paper/2211.07067/) (2022)

## Built on by (most cited first, 9 of 9)

- [Contextualizing Search Queries In-Context Learning for Conversational Rewriting with LLMs](https://mlsanity.com/paper/2502.15009/) (2025)
- [POET: Preference Optimization for Enhanced Text-to-Image Generation](https://mlsanity.com/paper/2510.12041/) (2025)
- [BudgetLeak: Membership Inference Attacks on RAG Systems via the Generation Budget Side Channel](https://mlsanity.com/paper/2511.12043/) (2025)
- [Connect the Dots: Knowledge Graph-Guided Crawler Attack on Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2601.15678/) (2026)
- [S3-R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Data](https://mlsanity.com/paper/2605.01248/) (2026)
- [A Survey of Query Optimization in Large Language Models](https://mlsanity.com/paper/2412.17558/) (2024)
- [SAGE: Strategy-Adaptive Generation Engine for Query Rewriting](https://mlsanity.com/paper/2506.19783/) (2025)
- [Rethinking RAG in Long Videos: What to Retrieve and How to Use It?](https://mlsanity.com/paper/2606.13141/) (2026)
- [E-MIA: Exam-Style Black-Box Membership Inference Attacks against RAG Systems](https://mlsanity.com/paper/2605.00955/) (2026)

## Abstract (arXiv)

In a real-world RAG system, the current query often involves spoken ellipses and ambiguous references from dialogue contexts, necessitating query rewriting to better describe user's information needs. However, traditional context-based rewriting has minimal enhancement on downstream generation tasks due to the lengthy process from query rewriting to response generation. Some researchers try to utilize reinforcement learning with generation feedback to assist the rewriter, but these sparse rewards provide little guidance in most cases, leading to unstable training and generation results. We find that user's needs are also reflected in the gold document, retrieved documents and ground truth. Therefore, by feeding back these multi-aspect dense rewards to query rewriting, more stable and satisfactory responses can be achieved. In this paper, we propose a novel query rewriting method MaFeRw, which improves RAG performance by integrating multi-aspect feedback from both the retrieval process and generated results. Specifically, we first use manual data to train a T5 model for the rewriter initialization. Next, we design three metrics as reinforcement learning feedback: the similarity between the rewritten query and the gold document, the ranking metrics, and ROUGE between the generation and the ground truth. Inspired by RLAIF, we train three kinds of reward models for the above metrics to achieve more efficient training. Finally, we combine the scores of these reward models as feedback, and use PPO algorithm to explore the optimal query rewriting strategy. Experimental results on two conversational RAG datasets demonstrate that MaFeRw achieves superior generation metrics and more stable training compared to baselines.