# Principled Evaluation with Human Labels: One Rater at a Time and Rater Equivalence

arXiv: https://arxiv.org/abs/2106.01254 · HTML: https://mlsanity.com/paper/2106.01254/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2021.

## Built on by (most cited first, 7 of 7)

- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels](https://mlsanity.com/paper/2405.05860/) (2024)
- [Venire: A Machine Learning-Guided Panel Review System for Community Content Moderation](https://mlsanity.com/paper/2410.23448/) (2024)
- [Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts](https://mlsanity.com/paper/2505.17222/) (2025)
- [Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors](https://mlsanity.com/paper/2410.13776/) (2024)
- [A Framework for Evaluating LLMs Under Task Indeterminacy](https://mlsanity.com/paper/2411.13760/) (2024)
- [How We Define Harm Impacts Data Annotations: Explaining How Annotators Distinguish Hateful, Offensive, and Toxic Comments](https://mlsanity.com/paper/2309.15827/) (2023)

## Abstract (arXiv)

In many classification tasks, there is no definitive ground truth, only human judgments that may disagree. We address two challenges that arise in such settings: (1) how to use human raters to score classifiers, and (2) how to use them for comparison benchmarks. For the first, the common practice is to score classifiers against the majority vote of an evaluation panel of several human raters. We argue that this is not justified when either of two properties fails: objectivity or equanimity. Instead, under a utility model appropriate for such settings, scoring against one rater at a time and averaging the scores across raters is a more principled approach. For the second, we introduce the concept of rater equivalence: the smallest number of human raters whose combined judgment matches the classifier's performance. We provide a provably optimal algorithm for combining benchmark panel labels, and demonstrate the framework through case studies.