# A Step Towards Mixture of Grader: Statistical Analysis of Existing Automatic Evaluation Metrics

arXiv: https://arxiv.org/abs/2410.10030 · HTML: https://mlsanity.com/paper/2410.10030/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 6 references that are in the map.
- First posted on arXiv 13 Oct 2024.

## Builds on (most cited first, 2 of 2)

- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering](https://mlsanity.com/paper/2307.16877/) (2023)

## Built on by (most cited first, 1 of 1)

- [Markovian ODE-guided scoring can assess the quality of offline reasoning traces in language models](https://mlsanity.com/paper/2603.01580/) (2026)

## Abstract (arXiv)

The explosion of open-sourced models and Question-Answering (QA) datasets emphasizes the importance of automated QA evaluation. We studied the statistics of the existing evaluation metrics for a better understanding of their limitations. By measuring the correlation coefficients of each evaluation metric concerning human-like evaluation score, we observed the following: (1) existing metrics have a high correlation among them concerning the question type (e.g., single word, single phrase, etc.), (2) no single metric can adequately estimate the human-like evaluation. As a potential solution, we discuss how a Mixture Of Grader could potentially improve the auto QA evaluator quality.