mlsanityData to 14 Oct 2026

QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs

arXiv 2205.12665 ↗First posted 25 May 2022cs.CLcited by 42

Not on the map. Why?

Abstract (arXiv)

Existing benchmarks for open-domain question answering (ODQA) typically focus on questions whose answers can be extracted from a single paragraph. By contrast, many natural questions, such as "What players were drafted by the Brooklyn Nets?" have a list of answers. Answering such questions requires retrieving and reading from many passages, in a large corpus. We introduce QAMPARI, an ODQA benchmark, where question answers are lists of entities, spread across many paragraphs. We created QAMPARI by (a) generating questions with multiple answers from Wikipedia's knowledge graph and tables, (b) automatically pairing answers with supporting evidence in Wikipedia paragraphs, and (c) manually paraphrasing questions and validating each answer. We train ODQA models from the retrieve-and-read family and find that QAMPARI is challenging in terms of both passage retrieval and answer generation, reaching an F1 score of 32.8 at best. Our results highlight the need for developing ODQA models that handle a broad range of question types, including single and multi-answer questions.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 29 of its 58 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

29 of 58 references.

References · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  10. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  12. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
17 more
  1. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  2. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  3. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  4. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  5. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  6. 2022Transformer Memory as a Differentiable Search Indexcited by 247
  7. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  8. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 179
  9. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  10. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  11. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 138
  12. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  13. 2019Understanding Dataset Design Choices for Multi-hop Reasoningcited by 50
  14. 2021Learning to Retrieve Passages without Supervisioncited by 46
  15. 2020Differentiable Reasoning over a Virtual Knowledge Basecited by 39
  16. 2021Database Reasoning Over Textcited by 20
  17. 2021Reasoning Over Virtual Knowledge Bases With Open Predicate Relationscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 11; 2024: 5; 2025: 15; 2026: 9.

42 papers.

Sort:

By area:

  1. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  2. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  3. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 56
  4. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52
  5. 2023Effective Large Language Model Adaptation for Improved Grounding and Citation Generationcited by 45
  6. 2023LitCab: Lightweight Language Model Calibration over Short- and Long-form Responsescited by 44
  7. 2023IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositionscited by 26
  8. 2024CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systemscited by 25
  9. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrievalcited by 20
  10. 2022RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answeringcited by 19
  11. 2023SEMQA: Semi-Extractive Multi-Source Question Answeringcited by 19
  12. 2023Towards Robust Temporal Reasoning of Large Language Models via a Multi-Hop QA Dataset and Pseudo-Instruction Tuningcited by 18
30 more
  1. 2024Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphscited by 17
  2. 2024Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?cited by 15
  3. 2023Answering Ambiguous Questions via Iterative Promptingcited by 13
  4. 2023Towards Reliable and Fluent Large Language Models: Incorporating Feedback Learning Loops in QA Systemscited by 10
  5. 2025Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Modelscited by 7
  6. 2025Generation-Time vs. Post-hoc Citation: A Holistic Evaluation of LLM Attributioncited by 6
  7. 2026SPADER: Step-wise Peer Advantage with Diversity-Aware Exploration Rewards for Multi-Answer Question Answeringcited by 6
  8. 2025Consensus or Conflict? Fine-Grained Evaluation of Conflicting Answers in Question-Answeringcited by 5
  9. 2025CompLLM: Compression for Long Context Q&Acited by 5
  10. 2025Beyond Single Embeddings: Capturing Diverse Targets with Multi-Query Retrievalcited by 5
  11. 2025InFact: Informativeness Alignment for Improved LLM Factualitycited by 3
  12. 2025Structured RAG for Answering Aggregative Questionscited by 3
  13. 2025NER Retriever: Zero-Shot Named Entity Retrieval with Type-Aware Embeddingscited by 2
  14. 2026Principled and Scalable Diversity-Aware Retrieval via Cardinality-Constrained Binary Quadratic Programmingcited by 2
  15. 2026AuthTrace: Diagnosing Evidence Construction in Thematically Dense Single-Author Corporacited by 2
  16. 2026CalVerT: Augmenting Agents with Calibrated Verifier Telemetry Improves Action and Learning in Knowledge-Intensive Taskscited by 2
  17. 2024Towards Robust Evaluation: A Comprehensive Taxonomy of Datasets and Metrics for Open Domain Question Answering in the Era of Large Language Modelscited by 1
  18. 2025Pretrained LLMs Learn Multiple Types of Uncertaintycited by 1
  19. 2025Inter-Passage Verification for Multi-evidence Multi-answer QAcited by 1
  20. 2025Equipping Retrieval-Augmented Large Language Models with Document Structure Awarenesscited by 1
  21. 2025Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generationcited by 1
  22. 2026RVR: Retrieve-Verify-Retrieve for Comprehensive Question Answeringcited by 1
  23. 2023Crafting In-context Examples according to LMs' Parametric Knowledgecited by 0
  24. 2025Jointly Generating and Attributing Answers using Logits of Document-Identifier Tokenscited by 0
  25. 2025LongRecall: A Structured Approach for Robust Recall Evaluation in Long-Form Textcited by 0
  26. 2026Quantifying Retriever-Generator Alignment in RAG with Local Explanationscited by 0
  27. 2025C2-Cite: Contextual-Aware Citation Generation for Attributed Large Language Modelscited by 0
  28. 2026Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructscited by 0
  29. 2026Detecting Knowledge Inconsistencies Across Text, Tables, and Knowledge Graphscited by 0
  30. 2026A Tree-based RAG Framework for Evidence-Intensive QA via Adaptive Planning and Topology-Aware Evidence Gatheringcited by 0