mlsanityData to 7 Oct 2026

What's in a Name? Answer Equivalence For Open-Domain Question Answering

arXiv 2109.05289 ↗First posted 11 Sep 2021cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

A flaw in QA evaluation is that annotations often only provide one gold answer. Thus, model predictions semantically equivalent to the answer but superficially different are considered incorrect. This work explores mining alias entities from knowledge bases and using them as additional gold answers (i.e., equivalent answers). We incorporate answers for two settings: evaluation with additional answers and model training with equivalent answers. We analyse three QA benchmarks: Natural Questions, TriviaQA, and SQuAD. Answer expansion increases the exact match score on all datasets for evaluation, while incorporating it helps model training over real-world datasets. We ensure the additional answers are valid through a human post hoc evaluation.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 11 Sep 2021.

Builds on

21 of 29 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  7. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  8. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  9. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  10. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  11. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  12. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
9 more
  1. 2019A Discrete Hard EM Approach for Weakly Supervised Question Answeringcited by 78
  2. 2019Answering Complex Open-domain Questions Through Iterative Query Generationcited by 60
  3. 2021NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learnedcited by 52
  4. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
  5. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  6. 2020BERT-QE: Contextualized Query Expansion for Document Re-rankingcited by 40
  7. 2019A BERT Baseline for the Natural Questionscited by 33
  8. 2021Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrievalcited by 25
  9. 2019What Question Answering can Learn from Trivia Nerdscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 7; 2024: 6; 2025: 0; 2026: 1.

21 papers.

Sort:

By area:

  1. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  2. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
  3. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  4. 2023Improving Automatic VQA Evaluation Using Large Language Modelscited by 37
  5. 2024PEDANTS: Cheap but Effective and Interpretable Answer Equivalencecited by 31
  6. 2022Conversational Question Answering on Heterogeneous Sourcescited by 26
  7. 2024Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering with Multi-Granularity Answerscited by 18
  8. 2024Accurate and Nuanced Open-QA Evaluation Through Textual Entailmentcited by 12
  9. 2021Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Frameworkcited by 11
  10. 2024Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationshipscited by 7
  11. 2023Learning Answer Generation using Supervision from Automatic Question Answering Evaluatorscited by 3
  12. 2022Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metricscited by 2
9 more
  1. 2023Reimagining Retrieval Augmented Language Models for Answering Queriescited by 2
  2. 2024Return of EM: Entity-driven Answer Set Expansion for QA Evaluationcited by 2
  3. 2022Evaluation of Semantic Answer Similarity Metricscited by 1
  4. 2022Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarkscited by 1
  5. 2023Mastering the ABCDs of Complex Questions: Answer-Based Claim Decomposition for Fine-grained Self-Evaluationcited by 1
  6. 2023SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative Referencescited by 1
  7. 2022'Tis but Thy Name: Semantic Question Answering Evaluation with 11M Names for 1M Entitiescited by 0
  8. 2024CFMatch: Aligning Automated Answer Equivalence Evaluation with Expert Judgments For Open-Domain Question Answeringcited by 0
  9. 2026Look Before You Leap: Factual Decoding with Internal Attribution Signalscited by 0