# SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References

arXiv: https://arxiv.org/abs/2309.12250 · HTML: https://mlsanity.com/paper/2309.12250/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Factual Error Correction for Abstractive Summarization Models](https://mlsanity.com/paper/2010.08712/) (2020)
- [TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection](https://mlsanity.com/paper/1911.04118/) (2019)
- [Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation](https://mlsanity.com/paper/2202.07654/) (2022)
- [NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned](https://mlsanity.com/paper/2101.00133/) (2021)
- [What's in a Name? Answer Equivalence For Open-Domain Question Answering](https://mlsanity.com/paper/2109.05289/) (2021)
- [Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task](https://mlsanity.com/paper/1806.03578/) (2018)
- [Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework](https://mlsanity.com/paper/2109.04817/) (2021)
- [Answer Generation for Retrieval-based Question Answering Systems](https://mlsanity.com/paper/2106.00955/) (2021)
- [Joint Models for Answer Verification in Question Answering Systems](https://mlsanity.com/paper/2107.04217/) (2021)
- [AVA: an Automatic eValuation Approach to Question Answering Systems](https://mlsanity.com/paper/2005.00705/) (2020)
- [Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection](https://mlsanity.com/paper/2205.10455/) (2022)
- [Knowledge Transfer from Answer Ranking to Answer Generation](https://mlsanity.com/paper/2210.12865/) (2022)
- [Double Retrieval and Ranking for Accurate Question Answering](https://mlsanity.com/paper/2201.05981/) (2022)
- [Learning Answer Generation using Supervision from Automatic Question Answering Evaluators](https://mlsanity.com/paper/2305.15344/) (2023)
- [Context-Aware Transformer Pre-Training for Answer Sentence Selection](https://mlsanity.com/paper/2305.15358/) (2023)

## Built on by (most cited first, 1 of 1)

- [Measuring Retrieval Complexity in Question Answering Systems](https://mlsanity.com/paper/2406.03592/) (2024)

## Abstract (arXiv)

Evaluation of QA systems is very challenging and expensive, with the most reliable approach being human annotations of correctness of answers for questions. Recent works (AVA, BEM) have shown that transformer LM encoder based similarity metrics transfer well for QA evaluation, but they are limited by the usage of a single correct reference answer. We propose a new evaluation metric: SQuArE (Sentence-level QUestion AnsweRing Evaluation), using multiple reference answers (combining multiple correct and incorrect references) for sentence-form QA. We evaluate SQuArE on both sentence-level extractive (Answer Selection) and generative (GenQA) QA systems, across multiple academic and industrial datasets, and show that it outperforms previous baselines and obtains the highest correlation with human annotations.