# AVA: an Automatic eValuation Approach to Question Answering Systems

arXiv: https://arxiv.org/abs/2005.00705 · HTML: https://mlsanity.com/paper/2005.00705/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2020.

## Built on by (most cited first, 10 of 10)

- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Multiple-Choice Question Generation: Towards an Automated Assessment Framework](https://mlsanity.com/paper/2209.11830/) (2022)
- [DeepNote: Note-Centric Deep Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.08821/) (2024)
- [Learning Answer Generation using Supervision from Automatic Question Answering Evaluators](https://mlsanity.com/paper/2305.15344/) (2023)
- [Measuring Retrieval Complexity in Question Answering Systems](https://mlsanity.com/paper/2406.03592/) (2024)
- [Return of EM: Entity-driven Answer Set Expansion for QA Evaluation](https://mlsanity.com/paper/2404.15650/) (2024)
- [Reference-based Weak Supervision for Answer Sentence Selection using Web Data](https://mlsanity.com/paper/2104.08943/) (2021)
- [SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References](https://mlsanity.com/paper/2309.12250/) (2023)
- [SPARC-RAG: Adaptive Sequential-Parallel Scaling with Context Management for Retrieval-Augmented Generation](https://mlsanity.com/paper/2602.00083/) (2026)
- [PrefRAG: Preference-Driven Multi-Source Retrieval Augmented Generation](https://mlsanity.com/paper/2411.00689/) (2024)

## Abstract (arXiv)

We introduce AVA, an automatic evaluation approach for Question Answering, which given a set of questions associated with Gold Standard answers, can estimate system Accuracy. AVA uses Transformer-based language models to encode question, answer, and reference text. This allows for effectively measuring the similarity between the reference and an automatic answer, biased towards the question semantics. To design, train and test AVA, we built multiple large training, development, and test sets on both public and industrial benchmarks. Our innovative solutions achieve up to 74.7% in F1 score in predicting human judgement for single answers. Additionally, AVA can be used to evaluate the overall system Accuracy with an RMSE, ranging from 0.02 to 0.09, depending on the availability of multiple references.