# PEDANTS: Cheap but Effective and Interpretable Answer Equivalence

arXiv: https://arxiv.org/abs/2402.11161 · HTML: https://mlsanity.com/paper/2402.11161/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 57 references that are in the map.
- First posted on arXiv 17 Feb 2024.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [Autoregressive Search Engines: Generating Substrings as Document Identifiers](https://mlsanity.com/paper/2204.10628/) (2022)
- [A Critical Evaluation of Evaluations for Long-form Question Answering](https://mlsanity.com/paper/2305.18201/) (2023)
- [Coreference Resolution as Query-based Span Prediction](https://mlsanity.com/paper/1911.01746/) (2019)
- [ChatQA: Surpassing GPT-4 on Conversational QA and RAG](https://mlsanity.com/paper/2401.10225/) (2024)
- [Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLP](https://mlsanity.com/paper/2109.06598/) (2021)
- [IROF: a low resource evaluation metric for explanation methods](https://mlsanity.com/paper/2003.08747/) (2020)
- [MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics](https://mlsanity.com/paper/2010.03636/) (2020)
- [NLProlog: Reasoning with Weak Unification for Question Answering in Natural Language](https://mlsanity.com/paper/1906.06187/) (2019)
- [Merging Generated and Retrieved Knowledge for Open-Domain QA](https://mlsanity.com/paper/2310.14393/) (2023)
- [What's in a Name? Answer Equivalence For Open-Domain Question Answering](https://mlsanity.com/paper/2109.05289/) (2021)
- [Improving QA Model Performance with Cartographic Inoculation](https://mlsanity.com/paper/2401.17498/) (2024)

## Built on by (most cited first, 30 of 31)

- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above](https://mlsanity.com/paper/2502.14127/) (2025)
- [A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges](https://mlsanity.com/paper/2501.02189/) (2025)
- [Answer Matching Outperforms Multiple Choice for Language Model Evaluation](https://mlsanity.com/paper/2507.02856/) (2025)
- ["You Gotta be a Doctor, Lin": An Investigation of Name-Based Bias of Large Language Models in Employment Recommendations](https://mlsanity.com/paper/2406.12232/) (2024)
- [VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding](https://mlsanity.com/paper/2505.01481/) (2025)
- [JEV-as-a-Judge: Accept When Confident, Escalate When Unsure](https://mlsanity.com/paper/2609.26550/) (2026)
- [SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinement](https://mlsanity.com/paper/2409.19242/) (2024)
- [CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding](https://mlsanity.com/paper/2503.10688/) (2025)
- [Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?](https://mlsanity.com/paper/2410.15512/) (2024)
- [MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs](https://mlsanity.com/paper/2502.00997/) (2025)
- [Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios](https://mlsanity.com/paper/2406.05322/) (2024)
- [Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation](https://mlsanity.com/paper/2506.15068/) (2025)
- [Mind the Gap! Static and Interactive Evaluations of Large Audio Models](https://mlsanity.com/paper/2502.15919/) (2025)
- [From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges](https://mlsanity.com/paper/2601.08654/) (2026)
- [A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users](https://mlsanity.com/paper/2509.18632/) (2025)
- [Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2406.13114/) (2024)
- [GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration](https://mlsanity.com/paper/2502.19684/) (2025)
- [Compositional Coordination for Multi-Robot Teams with Large Language Models](https://mlsanity.com/paper/2507.16068/) (2025)
- [Large Language Models Struggle to Describe the Haystack without Human Help: Human-in-the-loop Evaluation of Topic Models](https://mlsanity.com/paper/2502.14748/) (2025)
- [Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators](https://mlsanity.com/paper/2503.06778/) (2025)
- [Principled Content Selection to Generate Diverse and Personalized Multi-Document Summaries](https://mlsanity.com/paper/2505.21859/) (2025)
- [MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs](https://mlsanity.com/paper/2506.15215/) (2025)
- [Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision](https://mlsanity.com/paper/2603.07025/) (2026)
- [AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QA](https://mlsanity.com/paper/2604.21766/) (2026)
- [AuRA: Internalizing Audio Understanding into LLMs as LoRA](https://mlsanity.com/paper/2606.11033/) (2026)
- [On the Influence of Gender and Race in Romantic Relationship Prediction from Large Language Models](https://mlsanity.com/paper/2410.03996/) (2024)
- [RAG-BioQA: A Retrieval-Augmented Generation Framework for Long-Form Biomedical Question Answering](https://mlsanity.com/paper/2510.01612/) (2025)
- [Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment](https://mlsanity.com/paper/2605.00022/) (2026)
- [AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?](https://mlsanity.com/paper/2605.28255/) (2026)

## Abstract (arXiv)

Question answering (QA) can only make progress if we know if an answer is correct, but current answer correctness (AC) metrics struggle with verbose, free-form answers from large language models (LLMs). There are two challenges with current short-form QA evaluations: a lack of diverse styles of evaluation data and an over-reliance on expensive and slow LLMs. LLM-based scorers correlate better with humans, but this expensive task has only been tested on limited QA datasets. We rectify these issues by providing rubrics and datasets for evaluating machine QA adopted from the Trivia community. We also propose an efficient, and interpretable QA evaluation that is more stable than an exact match and neural methods(BERTScore).