mlsanityData to 7 Oct 2026

PEDANTS: Cheap but Effective and Interpretable Answer Equivalence

arXiv 2402.11161 ↗First posted 17 Feb 2024cs.CL cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

Question answering (QA) can only make progress if we know if an answer is correct, but current answer correctness (AC) metrics struggle with verbose, free-form answers from large language models (LLMs). There are two challenges with current short-form QA evaluations: a lack of diverse styles of evaluation data and an over-reliance on expensive and slow LLMs. LLM-based scorers correlate better with humans, but this expensive task has only been tested on limited QA datasets. We rectify these issues by providing rubrics and datasets for evaluating machine QA adopted from the Trivia community. We also propose an efficient, and interpretable QA evaluation that is more stable than an exact match and neural methods(BERTScore).

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 28 of its 57 references that are in the map. First posted on arXiv 17 Feb 2024.

Builds on

28 of 57 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
16 more
  1. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  2. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  3. 2018CoQA: A Conversational Question Answering Challengecited by 674
  4. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  5. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  6. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
  7. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  8. 2019Coreference Resolution as Query-based Span Predictioncited by 91
  9. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
  10. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  11. 2020IROF: a low resource evaluation metric for explanation methodscited by 45
  12. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  13. 2019NLProlog: Reasoning with Weak Unification for Question Answering in Natural Languagecited by 36
  14. 2023Merging Generated and Retrieved Knowledge for Open-Domain QAcited by 36
  15. 2021What's in a Name? Answer Equivalence For Open-Domain Question Answeringcited by 21
  16. 2024Improving QA Model Performance with Cartographic Inoculationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 16; 2026: 8.

31 papers.

Sort:

By area:

  1. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  2. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  3. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  4. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 37
  5. 2024"You Gotta be a Doctor, Lin": An Investigation of Name-Based Bias of Large Language Models in Employment Recommendationscited by 33
  6. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  7. 2024SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinementcited by 15
  8. 2025CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understandingcited by 14
  9. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 14
  10. 2024Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?cited by 13
  11. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEscited by 13
  12. 2024Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarioscited by 12
19 more
  1. 2025Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generationcited by 11
  2. 2025Mind the Gap! Static and Interactive Evaluations of Large Audio Modelscited by 8
  3. 2025A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Userscited by 6
  4. 2026From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judgescited by 6
  5. 2024Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillationcited by 5
  6. 2025GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibrationcited by 5
  7. 2025Compositional Coordination for Multi-Robot Teams with Large Language Modelscited by 3
  8. 2025Large Language Models Struggle to Describe the Haystack without Human Help: Human-in-the-loop Evaluation of Topic Modelscited by 2
  9. 2025Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotatorscited by 1
  10. 2025Principled Content Selection to Generate Diverse and Personalized Multi-Document Summariescited by 1
  11. 2025MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMscited by 1
  12. 2026Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervisioncited by 1
  13. 2026AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QAcited by 1
  14. 2026AuRA: Internalizing Audio Understanding into LLMs as LoRAcited by 1
  15. 2024On the Influence of Gender and Race in Romantic Relationship Prediction from Large Language Modelscited by 0
  16. 2025RAG-BioQA: A Retrieval-Augmented Generation Framework for Long-Form Biomedical Question Answeringcited by 0
  17. 2026Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignmentcited by 0
  18. 2026AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?cited by 0
  19. 2026V=akQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answeringcited by 0