# What's in a Name? Answer Equivalence For Open-Domain Question Answering

arXiv: https://arxiv.org/abs/2109.05289 · HTML: https://mlsanity.com/paper/2109.05289/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 11 Sep 2021.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Generation-Augmented Retrieval for Open-domain Question Answering](https://mlsanity.com/paper/2009.08553/) (2020)
- [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://mlsanity.com/paper/1911.10470/) (2019)
- [A Discrete Hard EM Approach for Weakly Supervised Question Answering](https://mlsanity.com/paper/1909.04849/) (2019)
- [Answering Complex Open-domain Questions Through Iterative Query Generation](https://mlsanity.com/paper/1910.07000/) (2019)
- [NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned](https://mlsanity.com/paper/2101.00133/) (2021)
- [Semantic Answer Similarity for Evaluating Question Answering Models](https://mlsanity.com/paper/2108.06130/) (2021)
- [MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics](https://mlsanity.com/paper/2010.03636/) (2020)
- [BERT-QE: Contextualized Query Expansion for Document Re-ranking](https://mlsanity.com/paper/2009.07258/) (2020)
- [A BERT Baseline for the Natural Questions](https://mlsanity.com/paper/1901.08634/) (2019)
- [Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval](https://mlsanity.com/paper/2104.05883/) (2021)
- [What Question Answering can Learn from Trivia Nerds](https://mlsanity.com/paper/1910.14464/) (2019)

## Built on by (most cited first, 21 of 21)

- [Benchmarking Foundation Models with Language-Model-as-an-Examiner](https://mlsanity.com/paper/2306.04181/) (2023)
- [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://mlsanity.com/paper/2305.06984/) (2023)
- [Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation](https://mlsanity.com/paper/2202.07654/) (2022)
- [Improving Automatic VQA Evaluation Using Large Language Models](https://mlsanity.com/paper/2310.02567/) (2023)
- [PEDANTS: Cheap but Effective and Interpretable Answer Equivalence](https://mlsanity.com/paper/2402.11161/) (2024)
- [Conversational Question Answering on Heterogeneous Sources](https://mlsanity.com/paper/2204.11677/) (2022)
- [Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering with Multi-Granularity Answers](https://mlsanity.com/paper/2401.04695/) (2024)
- [Accurate and Nuanced Open-QA Evaluation Through Textual Entailment](https://mlsanity.com/paper/2405.16702/) (2024)
- [Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Framework](https://mlsanity.com/paper/2110.08544/) (2021)
- [Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationships](https://mlsanity.com/paper/2407.12543/) (2024)
- [Learning Answer Generation using Supervision from Automatic Question Answering Evaluators](https://mlsanity.com/paper/2305.15344/) (2023)
- [Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics](https://mlsanity.com/paper/2204.10206/) (2022)
- [Reimagining Retrieval Augmented Language Models for Answering Queries](https://mlsanity.com/paper/2306.01061/) (2023)
- [Return of EM: Entity-driven Answer Set Expansion for QA Evaluation](https://mlsanity.com/paper/2404.15650/) (2024)
- [Evaluation of Semantic Answer Similarity Metrics](https://mlsanity.com/paper/2206.12664/) (2022)
- [Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarks](https://mlsanity.com/paper/2210.05038/) (2022)
- [Mastering the ABCDs of Complex Questions: Answer-Based Claim Decomposition for Fine-grained Self-Evaluation](https://mlsanity.com/paper/2305.14750/) (2023)
- [SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References](https://mlsanity.com/paper/2309.12250/) (2023)
- ['Tis but Thy Name: Semantic Question Answering Evaluation with 11M Names for 1M Entities](https://mlsanity.com/paper/2202.13581/) (2022)
- [CFMatch: Aligning Automated Answer Equivalence Evaluation with Expert Judgments For Open-Domain Question Answering](https://mlsanity.com/paper/2401.13170/) (2024)
- [Look Before You Leap: Factual Decoding with Internal Attribution Signals](https://mlsanity.com/paper/2609.15745/) (2026)

## Abstract (arXiv)

A flaw in QA evaluation is that annotations often only provide one gold answer. Thus, model predictions semantically equivalent to the answer but superficially different are considered incorrect. This work explores mining alias entities from knowledge bases and using them as additional gold answers (i.e., equivalent answers). We incorporate answers for two settings: evaluation with additional answers and model training with equivalent answers. We analyse three QA benchmarks: Natural Questions, TriviaQA, and SQuAD. Answer expansion increases the exact match score on all datasets for evaluation, while incorporating it helps model training over real-world datasets. We ensure the additional answers are valid through a human post hoc evaluation.