# ANTIQUE: A Non-Factoid Question Answering Benchmark

arXiv: https://arxiv.org/abs/1905.08957 · HTML: https://mlsanity.com/paper/1905.08957/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2019.

## Built on by (most cited first, 24 of 24)

- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [Simplified Data Wrangling with ir_datasets](https://mlsanity.com/paper/2103.02280/) (2021)
- [Are Large Language Models Good at Utility Judgments?](https://mlsanity.com/paper/2403.19216/) (2024)
- [Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction](https://mlsanity.com/paper/2203.13088/) (2022)
- [Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2407.06992/) (2024)
- [Benchmarking LLM-based Relevance Judgment Methods](https://mlsanity.com/paper/2504.12558/) (2025)
- [AnswerSumm: A Manually-Curated Dataset and Pipeline for Answer Summarization](https://mlsanity.com/paper/2111.06474/) (2021)
- [Who's Your Judge? On the Detectability of LLM-Generated Judgments](https://mlsanity.com/paper/2509.25154/) (2025)
- [LINKAGE: Listwise Ranking among Varied-Quality References for Non-Factoid QA Evaluation via LLMs](https://mlsanity.com/paper/2409.14744/) (2024)
- [Multi-Perspective Abstractive Answer Summarization](https://mlsanity.com/paper/2104.08536/) (2021)
- [UIT-ViCoV19QA: A Dataset for COVID-19 Community-based Question Answering on Vietnamese Language](https://mlsanity.com/paper/2209.06668/) (2022)
- [An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMs](https://mlsanity.com/paper/2406.11290/) (2024)
- [Batch-Softmax Contrastive Loss for Pairwise Sentence Scoring Tasks](https://mlsanity.com/paper/2110.15725/) (2021)
- [Long-context Non-factoid Question Answering in Indic Languages](https://mlsanity.com/paper/2504.13615/) (2025)
- [Learning to Rank Question Answer Pairs with Bilateral Contrastive Data Augmentation](https://mlsanity.com/paper/2106.11096/) (2021)
- [GQE-PRF: Generative Query Expansion with Pseudo-Relevance Feedback](https://mlsanity.com/paper/2108.06010/) (2021)
- [Typed-RAG: Type-Aware Decomposition of Non-Factoid Questions for Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.15879/) (2025)
- [MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs](https://mlsanity.com/paper/2506.15215/) (2025)
- [Explainability of Text Processing and Retrieval Methods: A Survey](https://mlsanity.com/paper/2212.07126/) (2022)
- [Plan-and-Refine: Diverse and Comprehensive Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.07794/) (2025)
- [A model and package for German ColBERT](https://mlsanity.com/paper/2504.20083/) (2025)
- [How Do LLM-Generated Texts Impact Term-Based Retrieval Models?](https://mlsanity.com/paper/2508.17715/) (2025)
- [Inferential Question Answering](https://mlsanity.com/paper/2602.01239/) (2026)
- [OPERA: Online Data Pruning for Efficient Retrieval Model Adaptation](https://mlsanity.com/paper/2603.17205/) (2026)

## Abstract (arXiv)

Considering the widespread use of mobile and voice search, answer passage retrieval for non-factoid questions plays a critical role in modern information retrieval systems. Despite the importance of the task, the community still feels the significant lack of large-scale non-factoid question answering collections with real questions and comprehensive relevance judgments. In this paper, we develop and release a collection of 2,626 open-domain non-factoid questions from a diverse set of categories. The dataset, called ANTIQUE, contains 34,011 manual relevance annotations. The questions were asked by real users in a community question answering service, i.e., Yahoo! Answers. Relevance judgments for all the answers to each question were collected through crowdsourcing. To facilitate further research, we also include a brief analysis of the data as well as baseline results on both classical and recently developed neural IR models.