# Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators

arXiv: https://arxiv.org/abs/2111.13057 · HTML: https://mlsanity.com/paper/2111.13057/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2021.

## Built on by (most cited first, 30 of 34)

- [REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark](https://mlsanity.com/paper/2502.12342/) (2025)
- [NevIR: Negation in Neural Information Retrieval](https://mlsanity.com/paper/2305.07614/) (2023)
- [Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Method](https://mlsanity.com/paper/2308.09861/) (2023)
- [reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs](https://mlsanity.com/paper/2503.11751/) (2025)
- [CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typos](https://mlsanity.com/paper/2204.00716/) (2022)
- [Towards Imperceptible Document Manipulations against Neural Ranking Models](https://mlsanity.com/paper/2305.01860/) (2023)
- [Agent4Ranking: Semantic Robust Ranking via Personalized Query Rewriting Using Multi-agent LLM](https://mlsanity.com/paper/2312.15450/) (2023)
- [Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2407.06992/) (2024)
- [Out of Style: RAG's Fragility to Linguistic Variation](https://mlsanity.com/paper/2504.08231/) (2025)
- [Investigating the Robustness of Retrieval-Augmented Generation at the Query Level](https://mlsanity.com/paper/2507.06956/) (2025)
- [A Study on FGSM Adversarial Training for Neural Retrieval](https://mlsanity.com/paper/2301.10576/) (2023)
- [On the Robustness of Generative Retrieval Models: An Out-of-Distribution Perspective](https://mlsanity.com/paper/2306.12756/) (2023)
- [A Study on the Efficiency and Generalization of Light Hybrid Retrievers](https://mlsanity.com/paper/2210.01371/) (2022)
- [Generative Information Retrieval Evaluation](https://mlsanity.com/paper/2404.08137/) (2024)
- [Improving Content Retrievability in Search with Controllable Query Generation](https://mlsanity.com/paper/2303.11648/) (2023)
- [Typo-Robust Representation Learning for Dense Retrieval](https://mlsanity.com/paper/2306.10348/) (2023)
- [Noise-Robust Dense Retrieval via Contrastive Alignment Post Training](https://mlsanity.com/paper/2304.03401/) (2023)
- [Benchmark Granularity and Model Robustness for Image-Text Retrieval](https://mlsanity.com/paper/2407.15239/) (2024)
- [Defense of Adversarial Ranking Attack in Text Retrieval: Benchmark and Baseline via Detection](https://mlsanity.com/paper/2307.16816/) (2023)
- [How You Ask Matters! Adaptive RAG Robustness to Query Variations](https://mlsanity.com/paper/2604.10745/) (2026)
- [Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval](https://mlsanity.com/paper/2604.18360/) (2026)
- [Positional Bias in Long-Document Ranking: Impact, Assessment, and Mitigation](https://mlsanity.com/paper/2207.01262/) (2022)
- [Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcity](https://mlsanity.com/paper/2308.00177/) (2023)
- [Interventional Speech Noise Injection for ASR Generalizable Spoken Language Understanding](https://mlsanity.com/paper/2410.15609/) (2024)
- [Second SIGIR Workshop on Simulations for Information Access (Sim4IA 2025)](https://mlsanity.com/paper/2505.11687/) (2025)
- [Ontology-Guided Query Expansion for Biomedical Document Retrieval using Large Language Models](https://mlsanity.com/paper/2508.11784/) (2025)
- [On the Robustness of LLM-Based Dense Retrievers: A Systematic Analysis of Generalizability and Stability](https://mlsanity.com/paper/2604.16576/) (2026)
- [Lost in Decoding? Reproducing and Stress-Testing the Look-Ahead Prior in Generative Retrieval](https://mlsanity.com/paper/2604.23396/) (2026)
- [Hypencoder Revisited: Reproducibility and Analysis of Non-Linear Scoring for First-Stage Retrieval](https://mlsanity.com/paper/2604.27037/) (2026)
- [Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems](https://mlsanity.com/paper/2609.08887/) (2026)

## Abstract (arXiv)

Heavily pre-trained transformers for language modelling, such as BERT, have shown to be remarkably effective for Information Retrieval (IR) tasks, typically applied to re-rank the results of a first-stage retrieval model. IR benchmarks evaluate the effectiveness of retrieval pipelines based on the premise that a single query is used to instantiate the underlying information need. However, previous research has shown that (I) queries generated by users for a fixed information need are extremely variable and, in particular, (II) neural models are brittle and often make mistakes when tested with modified inputs. Motivated by those observations we aim to answer the following question: how robust are retrieval pipelines with respect to different variations in queries that do not change the queries' semantics? In order to obtain queries that are representative of users' querying variability, we first created a taxonomy based on the manual annotation of transformations occurring in a dataset (UQV100) of user-created query variations. For each syntax-changing category of our taxonomy, we employed different automatic methods that when applied to a query generate a query variation. Our experimental results across two datasets for two IR tasks reveal that retrieval pipelines are not robust to these query variations, with effectiveness drops of ≈20% on average. The code and datasets are available at https://github.com/Guzpenha/queryvariationgenerators.