# The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP

arXiv: https://arxiv.org/abs/2103.09710 · HTML: https://mlsanity.com/paper/2103.09710/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 40 references that are in the map.
- First posted on arXiv 17 Mar 2021.

## Builds on (most cited first, 4 of 4)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [With Little Power Comes Great Responsibility](https://mlsanity.com/paper/2010.06595/) (2020)

## Built on by (most cited first, 19 of 19)

- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [MAUVE Scores for Generative Models: Theory and Practice](https://mlsanity.com/paper/2212.14578/) (2022)
- [Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cards](https://mlsanity.com/paper/2108.07374/) (2021)
- [Where to start? Analyzing the potential value of intermediate models](https://mlsanity.com/paper/2211.00107/) (2022)
- [Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP](https://mlsanity.com/paper/2305.01633/) (2023)
- [A Review of Human Evaluation for Style Transfer](https://mlsanity.com/paper/2106.04747/) (2021)
- [Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization](https://mlsanity.com/paper/2212.10397/) (2022)
- [The Future of Open Human Feedback](https://mlsanity.com/paper/2408.16961/) (2024)
- [Quantified Reproducibility Assessment of NLP Results](https://mlsanity.com/paper/2204.05961/) (2022)
- [Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation](https://mlsanity.com/paper/2406.07935/) (2024)
- [Promoting the Responsible Development of Speech Datasets for Mental Health and Neurological Disorders Research](https://mlsanity.com/paper/2406.04116/) (2024)
- [QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2509.25664/) (2025)
- [Quantifying Reproducibility in NLP and ML](https://mlsanity.com/paper/2109.01211/) (2021)
- [Evaluating NLG systems: A brief introduction](https://mlsanity.com/paper/2303.16742/) (2023)
- [With a Little Help from the Authors: Reproducing Human Evaluation of an MT Error Detector](https://mlsanity.com/paper/2308.06527/) (2023)
- [HEDS 3.0: The Human Evaluation Data Sheet Version 3.0](https://mlsanity.com/paper/2412.07940/) (2024)
- [Natural Language Generation](https://mlsanity.com/paper/2503.16728/) (2025)
- [JUDGEBERT: Assessing Legal Meaning Preservation Between Sentences](https://mlsanity.com/paper/2508.16870/) (2025)

## Abstract (arXiv)

This paper introduces the Human Evaluation Datasheet, a template for recording the details of individual human evaluation experiments in Natural Language Processing (NLP). Originally taking inspiration from seminal papers by Bender and Friedman (2018), Mitchell et al. (2019), and Gebru et al. (2020), the Human Evaluation Datasheet is intended to facilitate the recording of properties of human evaluations in sufficient detail, and with sufficient standardisation, to support comparability, meta-evaluation, and reproducibility tests.