# The Reasonable Effectiveness of Diverse Evaluation Data

arXiv: https://arxiv.org/abs/2301.09406 · HTML: https://mlsanity.com/paper/2301.09406/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2023.

## Built on by (most cited first, 10 of 10)

- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [Disentangling Perceptions of Offensiveness: Cultural and Moral Correlates](https://mlsanity.com/paper/2312.06861/) (2023)
- [Consensus and Subjectivity of Skin Tone Annotation for ML Fairness](https://mlsanity.com/paper/2305.09073/) (2023)
- [Scaling Laws Do Not Scale](https://mlsanity.com/paper/2307.03201/) (2023)
- [D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation](https://mlsanity.com/paper/2404.10857/) (2024)
- [Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Models](https://mlsanity.com/paper/2507.13383/) (2025)
- [Generalized People Diversity: Learning a Human Perception-Aligned Diversity Representation for People Images](https://mlsanity.com/paper/2401.14322/) (2024)
- [Beyond Multiple-Choice Accuracy: Real-World Challenges of Implementing Large Language Models in Healthcare](https://mlsanity.com/paper/2410.18460/) (2024)
- [Reward Models are Metrics in a Trench Coat](https://mlsanity.com/paper/2510.03231/) (2025)
- [Differentially Private Preference Data Synthesis for Large Language Model Alignment](https://mlsanity.com/paper/2605.30808/) (2026)

## Abstract (arXiv)

In this paper, we present findings from an semi-experimental exploration of rater diversity and its influence on safety annotations of conversations generated by humans talking to a generative AI-chat bot. We find significant differences in judgments produced by raters from different geographic regions and annotation platforms, and correlate these perspectives with demographic sub-groups. Our work helps define best practices in model development -- specifically human evaluation of generative models -- on the backdrop of growing work on sociotechnical AI evaluations.