# Semantic Answer Similarity for Evaluating Question Answering Models

arXiv: https://arxiv.org/abs/2108.06130 · HTML: https://mlsanity.com/paper/2108.06130/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 13 Aug 2021.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [Look at the First Sentence: Position Bias in Question Answering](https://mlsanity.com/paper/2004.14602/) (2020)
- [Towards a Better Metric for Evaluating Question Generation Systems](https://mlsanity.com/paper/1808.10192/) (2018)
- [NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned](https://mlsanity.com/paper/2101.00133/) (2021)
- [Can NLI Models Verify QA Systems' Predictions?](https://mlsanity.com/paper/2104.08731/) (2021)
- [GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval](https://mlsanity.com/paper/2104.12741/) (2021)
- [Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task](https://mlsanity.com/paper/1806.03578/) (2018)
- [ReQA: An Evaluation for End-to-End Answer Retrieval Models](https://mlsanity.com/paper/1907.04780/) (2019)
- [What do we expect from Multiple-choice QA Systems?](https://mlsanity.com/paper/2011.10647/) (2020)
- [Comparative Analysis of Neural QA models on SQuAD](https://mlsanity.com/paper/1806.06972/) (2018)

## Built on by (most cited first, 30 of 47)

- [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://mlsanity.com/paper/2305.06984/) (2023)
- [Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation](https://mlsanity.com/paper/2202.07654/) (2022)
- [Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training](https://mlsanity.com/paper/2406.00222/) (2024)
- [MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance](https://mlsanity.com/paper/2503.13509/) (2025)
- [Improving Automatic VQA Evaluation Using Large Language Models](https://mlsanity.com/paper/2310.02567/) (2023)
- [Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy](https://mlsanity.com/paper/2402.07270/) (2024)
- [What's in a Name? Answer Equivalence For Open-Domain Question Answering](https://mlsanity.com/paper/2109.05289/) (2021)
- [Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models](https://mlsanity.com/paper/2311.08106/) (2023)
- [Reassessing Extractive QA Datasets at Scale: LLM-as-a-Judge and In-Depth Analyses](https://mlsanity.com/paper/2504.11972/) (2025)
- [Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking](https://mlsanity.com/paper/2505.23495/) (2025)
- [ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models](https://mlsanity.com/paper/2408.07983/) (2024)
- [SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression](https://mlsanity.com/paper/2507.05633/) (2025)
- [PlugMed: Improving Specificity in Patient-Centered Medical Dialogue Generation using In-Context Learning](https://mlsanity.com/paper/2305.11508/) (2023)
- [Introducing a new hyper-parameter for RAG: Context Window Utilization](https://mlsanity.com/paper/2407.19794/) (2024)
- [Evaluating Quality of Answers for Retrieval-Augmented Generation: A Strong LLM Is All You Need](https://mlsanity.com/paper/2406.18064/) (2024)
- [MedKP: Medical Dialogue with Knowledge Enhancement and Clinical Pathway Encoding](https://mlsanity.com/paper/2403.06611/) (2024)
- [Self-Improving Customer Review Response Generation Based on LLMs](https://mlsanity.com/paper/2405.03845/) (2024)
- [Selectively Answering Visual Questions](https://mlsanity.com/paper/2406.00980/) (2024)
- [A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs](https://mlsanity.com/paper/2501.13620/) (2025)
- [SCAI-QReCC Shared Task on Conversational Question Answering](https://mlsanity.com/paper/2201.11094/) (2022)
- [Can an AI Win Ghana's National Science and Maths Quiz? An AI Grand Challenge for Education](https://mlsanity.com/paper/2301.13089/) (2023)
- [Question Answering under Temporal Conflict: Evaluating and Organizing Evolving Knowledge with LLMs](https://mlsanity.com/paper/2506.07270/) (2025)
- [Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation](https://mlsanity.com/paper/2602.17316/) (2026)
- [Exploring the Readiness of Prominent Small Language Models for the Democratization of Financial Literacy](https://mlsanity.com/paper/2410.07118/) (2024)
- [An evaluation of LLM code generation capabilities through graded exercises](https://mlsanity.com/paper/2410.16292/) (2024)
- [Familiarity: Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Data](https://mlsanity.com/paper/2412.10121/) (2024)
- [ABCD: All Biases Come Disguised](https://mlsanity.com/paper/2602.17445/) (2026)
- [Evaluation of medium-large Language Models at zero-shot closed book generative question answering](https://mlsanity.com/paper/2305.11991/) (2023)
- [Return of EM: Entity-driven Answer Set Expansion for QA Evaluation](https://mlsanity.com/paper/2404.15650/) (2024)
- [La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America](https://mlsanity.com/paper/2507.00999/) (2025)

## Abstract (arXiv)

The evaluation of question answering models compares ground-truth annotations with model predictions. However, as of today, this comparison is mostly lexical-based and therefore misses out on answers that have no lexical overlap but are still semantically similar, thus treating correct answers as false. This underestimation of the true performance of models hinders user acceptance in applications and complicates a fair comparison of different models. Therefore, there is a need for an evaluation metric that is based on semantics instead of pure string similarity. In this short paper, we present SAS, a cross-encoder-based metric for the estimation of semantic answer similarity, and compare it to seven existing metrics. To this end, we create an English and a German three-way annotated evaluation dataset containing pairs of answers along with human judgment of their semantic similarity, which we release along with an implementation of the SAS metric and the experiments. We find that semantic similarity metrics based on recent transformer models correlate much better with human judgment than traditional lexical similarity metrics on our two newly created datasets and one dataset from related work.