mlsanityData to 14 Oct 2026

Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators

arXiv 2111.13057 ↗First posted 25 Nov 2021cs.IRcited by 34

Not on the map. Why?

Abstract (arXiv)

Heavily pre-trained transformers for language modelling, such as BERT, have shown to be remarkably effective for Information Retrieval (IR) tasks, typically applied to re-rank the results of a first-stage retrieval model. IR benchmarks evaluate the effectiveness of retrieval pipelines based on the premise that a single query is used to instantiate the underlying information need. However, previous research has shown that (I) queries generated by users for a fixed information need are extremely variable and, in particular, (II) neural models are brittle and often make mistakes when tested with modified inputs. Motivated by those observations we aim to answer the following question: how robust are retrieval pipelines with respect to different variations in queries that do not change the queries' semantics? In order to obtain queries that are representative of users' querying variability, we first created a taxonomy based on the manual annotation of transformations occurring in a dataset (UQV100) of user-created query variations. For each syntax-changing category of our taxonomy, we employed different automatic methods that when applied to a query generate a query variation. Our experimental results across two datasets for two IR tasks reveal that retrieval pipelines are not robust to these query variations, with effectiveness drops of ≈20% on average. The code and datasets are available at https://github.com/Guzpenha/queryvariationgenerators.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 Nov 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 11; 2024: 4; 2025: 6; 2026: 10.

34 papers.

Sort:

By area:

  1. 2025REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmarkcited by 35
  2. 2023NevIR: Negation in Neural Information Retrievalcited by 29
  3. 2023Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Methodcited by 29
  4. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  5. 2022CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typoscited by 22
  6. 2023Towards Imperceptible Document Manipulations against Neural Ranking Modelscited by 18
  7. 2023Agent4Ranking: Semantic Robust Ranking via Personalized Query Rewriting Using Multi-agent LLMcited by 18
  8. 2024Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspectivecited by 15
  9. 2025Out of Style: RAG's Fragility to Linguistic Variationcited by 11
  10. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Levelcited by 11
  11. 2023A Study on FGSM Adversarial Training for Neural Retrievalcited by 10
  12. 2023On the Robustness of Generative Retrieval Models: An Out-of-Distribution Perspectivecited by 8
22 more
  1. 2022A Study on the Efficiency and Generalization of Light Hybrid Retrieverscited by 7
  2. 2024Generative Information Retrieval Evaluationcited by 7
  3. 2023Improving Content Retrievability in Search with Controllable Query Generationcited by 6
  4. 2023Typo-Robust Representation Learning for Dense Retrievalcited by 6
  5. 2023Noise-Robust Dense Retrieval via Contrastive Alignment Post Trainingcited by 5
  6. 2024Benchmark Granularity and Model Robustness for Image-Text Retrievalcited by 5
  7. 2023Defense of Adversarial Ranking Attack in Text Retrieval: Benchmark and Baseline via Detectioncited by 3
  8. 2026How You Ask Matters! Adaptive RAG Robustness to Query Variationscited by 3
  9. 2026Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrievalcited by 3
  10. 2022Positional Bias in Long-Document Ranking: Impact, Assessment, and Mitigationcited by 2
  11. 2023Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcitycited by 1
  12. 2024Interventional Speech Noise Injection for ASR Generalizable Spoken Language Understandingcited by 1
  13. 2025Second SIGIR Workshop on Simulations for Information Access (Sim4IA 2025)cited by 0
  14. 2025Ontology-Guided Query Expansion for Biomedical Document Retrieval using Large Language Modelscited by 0
  15. 2026On the Robustness of LLM-Based Dense Retrievers: A Systematic Analysis of Generalizability and Stabilitycited by 0
  16. 2026Lost in Decoding? Reproducing and Stress-Testing the Look-Ahead Prior in Generative Retrievalcited by 0
  17. 2026Hypencoder Revisited: Reproducibility and Analysis of Non-Linear Scoring for First-Stage Retrievalcited by 0
  18. 2026Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systemscited by 0
  19. 2026CORA: A Protocol for Diagnosing Boundary Robustness in Text-to-Audio Retrieval under Query Reformulationscited by 0
  20. 2026Retrieval Sensitivity to Identity Signals in Queriescited by 0
  21. 2026Lost in Conversation or Lost in Translation? Diagnosing Multi-Turn Degradation in RAGcited by 0
  22. 2026Lost in the Request: How Communication Variation Disrupts Retrieval and Action in Email Agentscited by 0