# QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations

arXiv: https://arxiv.org/abs/2305.11694 · HTML: https://mlsanity.com/paper/2305.11694/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 22 of 22)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://mlsanity.com/paper/2112.07916/) (2021)
- [Efficient Natural Language Response Suggestion for Smart Reply](https://mlsanity.com/paper/1705.00652/) (2017)
- [PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text](https://mlsanity.com/paper/1904.09537/) (2019)
- [Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases](https://mlsanity.com/paper/2011.07743/) (2020)
- [Deeper Text Understanding for IR with Contextual Neural Language Modeling](https://mlsanity.com/paper/1905.09217/) (2019)
- [A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutions](https://mlsanity.com/paper/2105.11644/) (2021)
- [Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures](https://mlsanity.com/paper/1805.06627/) (2018)
- [QED: A Framework and Dataset for Explanations in Question Answering](https://mlsanity.com/paper/2009.06354/) (2020)
- [Joint Passage Ranking for Diverse Multi-Answer Retrieval](https://mlsanity.com/paper/2104.08445/) (2021)
- [RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering](https://mlsanity.com/paper/2210.14353/) (2022)
- [Question Answering from Unstructured Text by Retrieval and Comprehension](https://mlsanity.com/paper/1703.08885/) (2017)

## Built on by (most cited first, 30 of 31)

- [Chain-of-Verification Reduces Hallucination in Large Language Models](https://mlsanity.com/paper/2309.11495/) (2023)
- [On the Theoretical Limitations of Embedding-Based Retrieval](https://mlsanity.com/paper/2508.21038/) (2025)
- [Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?](https://mlsanity.com/paper/2406.13121/) (2024)
- [MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertainty](https://mlsanity.com/paper/2408.06816/) (2024)
- [Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Art](https://mlsanity.com/paper/2403.16527/) (2024)
- [NERetrieve: Dataset for Next Generation Named Entity Recognition and Retrieval](https://mlsanity.com/paper/2310.14282/) (2023)
- [Atomic Self-Consistency for Better Long Form Generations](https://mlsanity.com/paper/2405.13131/) (2024)
- [BoolQuestions: Does Dense Retrieval Understand Boolean Logic in Language?](https://mlsanity.com/paper/2411.12235/) (2024)
- [Benchmarking Information Retrieval Models on Complex Retrieval Tasks](https://mlsanity.com/paper/2509.07253/) (2025)
- [SPADER: Step-wise Peer Advantage with Diversity-Aware Exploration Rewards for Multi-Answer Question Answering](https://mlsanity.com/paper/2606.00593/) (2026)
- [SetBERT: Enhancing Retrieval Performance for Boolean Logic and Set Operation Queries](https://mlsanity.com/paper/2406.17282/) (2024)
- [CompLLM: Compression for Long Context Q&A](https://mlsanity.com/paper/2509.19228/) (2025)
- [When Is Enough Not Enough? Illusory Completion in Search Agents](https://mlsanity.com/paper/2602.07549/) (2026)
- [Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents](https://mlsanity.com/paper/2603.18516/) (2026)
- [Retrieval Augmented Zero-Shot Text Classification](https://mlsanity.com/paper/2406.15241/) (2024)
- [A Comprehensive Taxonomy of Negation for NLP and Neural Retrievers](https://mlsanity.com/paper/2507.22337/) (2025)
- [Generating Tables from the Parametric Knowledge of Language Models](https://mlsanity.com/paper/2406.10922/) (2024)
- [LogiCoL: Logically-Informed Contrastive Learning for Set-based Dense Retrieval](https://mlsanity.com/paper/2505.19588/) (2025)
- [Structured RAG for Answering Aggregative Questions](https://mlsanity.com/paper/2511.08505/) (2025)
- [NER Retriever: Zero-Shot Named Entity Retrieval with Type-Aware Embeddings](https://mlsanity.com/paper/2509.04011/) (2025)
- [MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image](https://mlsanity.com/paper/2605.10616/) (2026)
- [RVR: Retrieve-Verify-Retrieve for Comprehensive Question Answering](https://mlsanity.com/paper/2602.18425/) (2026)
- [NSFL: A Post-Training Neuro-Symbolic Fuzzy Logic Framework for Boolean Operators in Neural Embeddings](https://mlsanity.com/paper/2604.10604/) (2026)
- [Crafting In-context Examples according to LMs' Parametric Knowledge](https://mlsanity.com/paper/2311.09579/) (2023)
- [A Geometric Approach to Personalized Recommendation with Set-Theoretic Constraints Using Box Embeddings](https://mlsanity.com/paper/2502.10875/) (2025)
- [Boolean-aware Attention for Dense Retrieval](https://mlsanity.com/paper/2503.01753/) (2025)
- [Evaluation of retrieval-based QA on QUEST-LOFT](https://mlsanity.com/paper/2511.06125/) (2025)
- [Tutorial on Reasoning for IR & IR for Reasoning](https://mlsanity.com/paper/2602.03640/) (2026)
- [Reproducing Complex Set-Compositional Information Retrieval](https://mlsanity.com/paper/2605.03824/) (2026)
- [LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning](https://mlsanity.com/paper/2609.38137/) (2026)

## Abstract (arXiv)

Formulating selective information needs results in queries that implicitly specify set operations, such as intersection, union, and difference. For instance, one might search for "shorebirds that are not sandpipers" or "science-fiction films shot in England". To study the ability of retrieval systems to meet such information needs, we construct QUEST, a dataset of 3357 natural language queries with implicit set operations, that map to a set of entities corresponding to Wikipedia documents. The dataset challenges models to match multiple constraints mentioned in queries with corresponding evidence in documents and correctly perform various set operations. The dataset is constructed semi-automatically using Wikipedia category names. Queries are automatically composed from individual categories, then paraphrased and further validated for naturalness and fluency by crowdworkers. Crowdworkers also assess the relevance of entities based on their documents and highlight attribution of query constraints to spans of document text. We analyze several modern retrieval systems, finding that they often struggle on such queries. Queries involving negation and conjunction are particularly challenging and systems are further challenged with combinations of these operations.