mlsanityData to 7 Oct 2026

Not All Queries Need Rewriting: When Prompt-Only LLM Refinement Helps and Hurts Dense Retrieval

arXiv 2603.13301 ↗First posted 2 Mar 2026cs.IR cs.AI cs.LGcited by 1

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Prompt-only, single-step LLM query rewriting, where a rewrite is generated from the query alone without retrieval feedback, is commonly used in production RAG pipelines, but its effect on dense retrieval is poorly understood. We present a systematic empirical study across three BEIR benchmarks, two dense retrievers, and multiple training configurations, and find strongly domain-dependent behavior: rewriting degrades nDCG@10 by 9.0 percent on FiQA, improves it by 5.1 percent on TREC-COVID, and has no significant effect on SciFact. We identify a consistent mechanism: degradations co-occur with reduced lexical alignment between rewritten queries and relevant documents, as rewriting replaces domain-specific terms in already well-matched queries. In contrast, improvements arise when rewriting shifts queries toward corpus-preferred terminology and resolves inconsistent nomenclature. Lexical substitution occurs in 95 percent of rewrites across all outcome groups, showing that effectiveness depends on the direction of substitution rather than substitution itself. We also study selective rewriting and find that simple feature-based gating can reduce worst-case regressions but does not reliably outperform never rewriting, with even oracle selection offering only modest gains. Overall, these results show that prompt-only rewriting can be harmful in well-optimized verticals and suggest that domain-adaptive post-training is a safer strategy when supervision or implicit feedback is available.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 24 of its 30 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

24 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 22

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2017Billion-scale similarity search with GPUscited by 2,477
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  5. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  6. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  7. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  8. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  9. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  10. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  11. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  12. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
10 more
  1. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  2. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  3. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  4. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  5. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  6. 2019Document Expansion by Query Predictioncited by 225
  7. 2023Query2doc: Query Expansion with Large Language Modelscited by 146
  8. 2023Query Expansion by Prompting Large Language Modelscited by 56
  9. 2018Pytrec_eval: An Extremely Fast Python Interface to trec_evalcited by 46
  10. 2024Retrieval Augmented Generation for Domain-specific Question Answeringcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Two Wrongs, No Right: Auditing Social-Desirability Bias in LLM Annotators for Computational Social Sciencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Better Together: Complementary Query Rewriting Under a Strong RAG Baseline
  2. 2024RaFe: Ranking Feedback Improves Query Rewriting for RAG
  3. 2024DMQR-RAG: Diverse Multi-Query Rewriting for RAG
  4. 2026Masking or Mitigating? Deconstructing the Impact of Query Rewriting on Retriever Biases in RAG
  5. 2023Query Rewriting for Retrieval-Augmented Large Language Models
  6. 2024MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language Models
  7. 2025GRAPE: Let GRPO Supervise Query Rewriting by Ranking for Retrieval
  8. 2026Sifei at SemEval-2026 Task 8: Hybrid Retrieval and Query Rewriting for Multi-Turn RAG