mlsanityData to 7 Oct 2026

Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation

arXiv 2202.07654 ↗First posted 15 Feb 2022cs.CL cs.LGcited by 66

Not on the map. Why?

Abstract (arXiv)

The predictions of question answering (QA)systems are typically evaluated against manually annotated finite sets of one or more answers. This leads to a coverage limitation that results in underestimating the true performance of systems, and is typically addressed by extending over exact match (EM) with pre-defined rules or with the token-level F1 measure. In this paper, we present the first systematic conceptual and data-driven analysis to examine the shortcomings of token-level equivalence measures. To this end, we define the asymmetric notion of answer equivalence (AE), accepting answers that are equivalent to or improve over the reference, and publish over 23k human judgments for candidates produced by multiple QA systems on SQuAD. Through a careful analysis of this data, we reveal and quantify several concrete limitations of the F1 measure, such as a false impression of graduality, or missing dependence on the question. Since collecting AE annotations for each evaluated model is expensive, we learn a BERT matching (BEM) measure to approximate this task. Being a simpler task than QA, we find BEM to provide significantly better AE approximations than F1, and to more accurately reflect the performance of systems. Finally, we demonstrate the practical utility of AE and BEM on the concrete application of minimal accurate prediction sets, reducing the number of required answers by up to x2.6.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 29 of its 54 references that are in the map. First posted on arXiv 15 Feb 2022.

Builds on

29 of 54 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  8. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  9. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  10. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  11. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  12. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
17 more
  1. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  2. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  3. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  4. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  5. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  6. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  7. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 131
  8. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  9. 2019Question Answering as an Automatic Evaluation Metric for News Article Summarizationcited by 63
  10. 2021NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learnedcited by 52
  11. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
  12. 2021Can NLI Models Verify QA Systems' Predictions?cited by 46
  13. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  14. 2020Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Taskcited by 25
  15. 2021What's in a Name? Answer Equivalence For Open-Domain Question Answeringcited by 21
  16. 2018Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Taskcited by 20
  17. 2019Where's My Head? Definition, Dataset and Models for Numeric Fused-Heads Identification and Resolutioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 16; 2024: 17; 2025: 19; 2026: 9.

66 papers.

Sort:

By area:

  1. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  2. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 161
  3. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  4. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 124
  5. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
  6. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 108
  7. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  8. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  9. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  10. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  11. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  12. 2024MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMscited by 45
38 more
  1. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44
  2. 2025MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistancecited by 38
  3. 2023Improving Automatic VQA Evaluation Using Large Language Modelscited by 37
  4. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 37
  5. 2024Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documentscited by 27
  6. 2024Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchycited by 26
  7. 2024Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generationcited by 25
  8. 2024Stratified Prediction-Powered Inference for Hybrid Language Model Evaluationcited by 24
  9. 2024Neptune: The Long Orbit to Benchmarking Long Video Understandingcited by 24
  10. 2023Evaluating and Modeling Attribution for Cross-Lingual Question Answeringcited by 21
  11. 2024Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answeringcited by 20
  12. 2024mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQAcited by 19
  13. 2023Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Modelscited by 17
  14. 2024Cross-modal Retrieval for Knowledge-based Visual Question Answeringcited by 13
  15. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  16. 2022Discord Questions: A Computational Approach To Diversity Analysis in News Coveragecited by 11
  17. 2025The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentationcited by 11
  18. 2023An Overview Of Temporal Commonsense Reasoning and Acquisitioncited by 10
  19. 2025A Study into Investigating Temporal Robustness of LLMscited by 10
  20. 2025Sherlock: Reliable and Efficient Agentic Workflow Executioncited by 10
  21. 2024Bayesian Prediction-Powered Inferencecited by 8
  22. 2023LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interactioncited by 7
  23. 2024Temporal Blind Spots in Large Language Modelscited by 7
  24. 2024HAMMR: HierArchical MultiModal React agents for generic VQAcited by 7
  25. 2025ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answeringcited by 7
  26. 2024Navigating the Landscape of Hint Generation Research: From the Past to the Futurecited by 6
  27. 2025Wrong Answers Can Also Be Useful: PlausibleQA -- A Large-Scale QA Dataset with Answer Plausibility Scorescited by 5
  28. 2025Describe Anything Model for Visual Question Answering on Text-rich Imagescited by 5
  29. 2023How (not) to ensemble LVLMs for VQAcited by 4
  30. 2024WikiHint: A Human-Annotated Dataset for Hint Ranking and Generationcited by 4
  31. 2025ImageChain: Advancing Sequential Image-to-Text Reasoning in Multimodal Large Language Modelscited by 4
  32. 2025LTRR: Learning To Rank Retrievers for LLMscited by 4
  33. 2025Revisiting NLI: Towards Cost-Effective and Human-Aligned Metrics for Evaluating LLMs in Question Answeringcited by 4
  34. 2025HGMEM: Hypergraph-based Working Memory to Improve Multi-step RAG for Long-Context Complex Relational Modelingcited by 4
  35. 2023DIFFQG: Generating Questions to Summarize Factual Changescited by 3
  36. 2023Learning Answer Generation using Supervision from Automatic Question Answering Evaluatorscited by 3
  37. 2025Recurrence Meets Transformers for Universal Multimodal Retrievalcited by 3
  38. 2023Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT for Mining Insights at Scalecited by 2

50 most cited of 66 All 66 (JSON)