# Unsupervised Evaluation for Question Answering with Transformers

arXiv: https://arxiv.org/abs/2010.03222 · HTML: https://mlsanity.com/paper/2010.03222/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2020.

## Built on by (most cited first, 2 of 2)

- [The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer](https://mlsanity.com/paper/2204.06457/) (2022)
- [An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems](https://mlsanity.com/paper/2602.21059/) (2026)

## Abstract (arXiv)

It is challenging to automatically evaluate the answer of a QA model at inference time. Although many models provide confidence scores, and simple heuristics can go a long way towards indicating answer correctness, such measures are heavily dataset-dependent and are unlikely to generalize. In this work, we begin by investigating the hidden representations of questions, answers, and contexts in transformer-based QA architectures. We observe a consistent pattern in the answer representations, which we show can be used to automatically evaluate whether or not a predicted answer span is correct. Our method does not require any labeled data and outperforms strong heuristic baselines, across 2 datasets and 7 domains. We are able to predict whether or not a model's answer is correct with 91.37% accuracy on SQuAD, and 80.7% accuracy on SubjQA. We expect that this method will have broad applications, e.g., in the semi-automatic development of QA datasets