mlsanityData to 7 Oct 2026

The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation

arXiv 2109.06835 ↗First posted 14 Sep 2021cs.CLcited by 96

Not on the map. Why?

Abstract (arXiv)

Recent text generation research has increasingly focused on open-ended domains such as story and poetry generation. Because models built for such tasks are difficult to evaluate automatically, most researchers in the space justify their modeling choices by collecting crowdsourced human judgments of text quality (e.g., Likert scores of coherence or grammaticality) from Amazon Mechanical Turk (AMT). In this paper, we first conduct a survey of 45 open-ended text generation papers and find that the vast majority of them fail to report crucial details about their AMT tasks, hindering reproducibility. We then run a series of story evaluation experiments with both AMT workers and English teachers and discover that even with strict qualification filters, AMT workers (unlike teachers) fail to distinguish between model-generated text and human-generated references. We show that AMT worker judgments improve when they are shown model-generated output alongside human-generated references, which enables the workers to better calibrate their ratings. Finally, interviews with the English teachers provide deeper insights into the challenges of the evaluation process, particularly when rating model-generated text.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Builds on 46 of its 90 references that are in the map. First posted on arXiv 14 Sep 2021.

Builds on

46 of 90 references.

References · 46

  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2018Hierarchical Neural Story Generationcited by 1,182
  3. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  4. 2020Evaluation of Text Generation: A Surveycited by 275
  5. 2019Automatic Detection of Generated Text is Easiest when Humans are Fooledcited by 260
  6. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 209
  7. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  8. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 181
  9. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  10. 2020With Little Power Comes Great Responsibilitycited by 138
  11. 2019Strategies for Structuring Story Generationcited by 124
  12. 2020Enabling Language Models to Fill in the Blankscited by 121
34 more
  1. 2018Learning to Write with Cooperative Discriminatorscited by 119
  2. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116
  3. 2020Neural CRF Model for Sentence Alignment in Text Simplificationcited by 106
  4. 2020PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Trackingcited by 104
  5. 2020Content Planning for Neural Story Generation with Aristotelian Rescoringcited by 92
  6. 2019Do Massively Pretrained Language Models Make Better Storytellers?cited by 90
  7. 2019Counterfactual Story Reasoning and Generationcited by 87
  8. 2019Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfercited by 81
  9. 2020MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Modelscited by 74
  10. 2020Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoningcited by 65
  11. 2019Evaluating Style Transfer for Textcited by 59
  12. 2018Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translationcited by 58
  13. 2019This Email Could Save Your Life: Introducing the Task of Email Subject Line Generationcited by 58
  14. 2020Neural Syntactic Preordering for Controlled Paraphrase Generationcited by 56
  15. 2020STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generationcited by 56
  16. 2018Towards Explainable NLP: A Generative Explanation Framework for Text Classificationcited by 55
  17. 2017Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learningcited by 44
  18. 2020Modeling Protagonist Emotions for Emotion-Aware Storytellingcited by 42
  19. 2019Pun Generation with Surprisecited by 37
  20. 2020Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioningcited by 36
  21. 2018Embedding Multimodal Relational Data for Knowledge Base Completioncited by 34
  22. 2020R3: Reverse, Retrieve, and Rank for Sarcasm Generation with Commonsense Knowledgecited by 32
  23. 2018Discourse-Aware Neural Rewards for Coherent Text Generationcited by 28
  24. 2020Sparse Text Generationcited by 24
  25. 2019Complexity-Weighted Loss and Diverse Reranking for Sentence Simplificationcited by 20
  26. 2019Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generationcited by 17
  27. 2019Improving Neural Story Generation by Targeted Common Sense Groundingcited by 16
  28. 2019Towards Generating Long and Coherent Text with Multi-Level Latent Variable Modelscited by 11
  29. 2019Visual Story Post-Editingcited by 10
  30. 2019Automated Chess Commentator Powered by Neural Chess Enginecited by 9
  31. 2019Generating Diverse Story Continuations with Controllable Semanticscited by 9
  32. 2019The Steep Road to Happily Ever After: An Analysis of Current Visual Storytelling Modelscited by 6
  33. 2020Learning to Generate Multiple Style Transfer Outputs for an Input Sentencecited by 2
  34. 2020Substance over Style: Document-Level Targeted Content Transfercited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 17; 2023: 28; 2024: 25; 2025: 16; 2026: 8.

96 papers.

Sort:

By area:

  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  2. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  4. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  5. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  6. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  7. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  8. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  9. 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
  10. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 168
  11. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  12. 2022Human heuristics for AI-generated language are flawedcited by 104
38 more
  1. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 98
  2. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  3. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  4. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  5. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  6. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  7. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  8. 2023Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generationcited by 66
  9. 2023The Next Chapter: A Study of Large Language Models in Storytellingcited by 58
  10. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  11. 2025People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated textcited by 39
  12. 2024KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Modelscited by 36
  13. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 34
  14. 2024LLM Harmony: Multi-Agent Communication for Problem Solvingcited by 34
  15. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 33
  16. 2022Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implicationscited by 32
  17. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  18. 2023I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphorscited by 31
  19. 2022Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russiancited by 29
  20. 2023Learning Personalized Alignment for Evaluating Open-ended Text Generationcited by 29
  21. 2025Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulationscited by 28
  22. 2021Analyzing Dynamic Adversarial Training Data in the Limitcited by 24
  23. 2024"They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversationscited by 24
  24. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  25. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  26. 2022How Do We Answer Complex Questions: Discourse Structure of Long-form Answerscited by 19
  27. 2022SNaC: Coherence Error Detection for Narrative Summarizationcited by 19
  28. 2024Understanding Figurative Meaning through Explainable Visual Entailmentcited by 19
  29. 2025Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboardscited by 19
  30. 2023Incorporating Worker Perspectives into MTurk Annotation Practices for NLPcited by 17
  31. 2024Lived Experience Not Found: LLMs Struggle to Align with Experts on Addressing Adverse Drug Reactions from Psychiatric Medication Usecited by 17
  32. 2025How Far Are AI Scientists from Changing the World?cited by 15
  33. 2023GROVE: A Retrieval-augmented Complex Story Generation Framework with A Forest of Evidencecited by 14
  34. 2023GIO: Gradient Information Optimization for Training Dataset Selectioncited by 13
  35. 2024Social Skill Training with Large Language Modelscited by 13
  36. 2024MoPS: Modular Story Premise Synthesis for Open-Ended Automatic Story Generationcited by 13
  37. 2024M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluationcited by 13
  38. 2021HydraSum: Disentangling Stylistic Features in Text Summarization using Multi-Decoder Modelscited by 12

50 most cited of 96 All 96 (JSON)