mlsanityData to 7 Oct 2026

The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP

arXiv 2103.09710 ↗First posted 17 Mar 2021cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

This paper introduces the Human Evaluation Datasheet, a template for recording the details of individual human evaluation experiments in Natural Language Processing (NLP). Originally taking inspiration from seminal papers by Bender and Friedman (2018), Mitchell et al. (2019), and Gebru et al. (2020), the Human Evaluation Datasheet is intended to facilitate the recording of properties of human evaluations in sufficient detail, and with sufficient standardisation, to support comparability, meta-evaluation, and reproducibility tests.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 4 of its 40 references that are in the map. First posted on arXiv 17 Mar 2021.

Builds on

4 of 40 references.

References · 4

  1. 2018Datasheets for Datasetscited by 1,371
  2. 2018Model Cards for Model Reportingcited by 1,068
  3. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  4. 2020With Little Power Comes Great Responsibilitycited by 138

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 5; 2023: 4; 2024: 4; 2025: 3.

19 papers.

Sort:

By area:

  1. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  2. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  3. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 34
  4. 2021Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cardscited by 24
  5. 2022Where to start? Analyzing the potential value of intermediate modelscited by 23
  6. 2023Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLPcited by 21
  7. 2021A Review of Human Evaluation for Style Transfercited by 19
  8. 2022Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarizationcited by 10
  9. 2024The Future of Open Human Feedbackcited by 10
  10. 2022Quantified Reproducibility Assessment of NLP Resultscited by 8
  11. 2024Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluationcited by 5
  12. 2024Promoting the Responsible Development of Speech Datasets for Mental Health and Neurological Disorders Researchcited by 3
7 more
  1. 2025QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairscited by 2
  2. 2021Quantifying Reproducibility in NLP and MLcited by 1
  3. 2023Evaluating NLG systems: A brief introductioncited by 0
  4. 2023With a Little Help from the Authors: Reproducing Human Evaluation of an MT Error Detectorcited by 0
  5. 2024HEDS 3.0: The Human Evaluation Data Sheet Version 3.0cited by 0
  6. 2025Natural Language Generationcited by 0
  7. 2025JUDGEBERT: Assessing Legal Meaning Preservation Between Sentencescited by 0