mlsanityData to 14 Oct 2026

Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation

arXiv 2208.11646 ↗First posted 24 Aug 2022cs.CLcited by 75

Not on the map. Why?

Abstract (arXiv)

Research on Automatic Story Generation (ASG) relies heavily on human and automatic evaluation. However, there is no consensus on which human evaluation criteria to use, and no analysis of how well automatic criteria correlate with them. In this paper, we propose to re-evaluate ASG evaluation. We introduce a set of 6 orthogonal and comprehensive human criteria, carefully motivated by the social sciences literature. We also present HANNA, an annotated dataset of 1,056 stories produced by 10 different ASG systems. HANNA allows us to quantitatively evaluate the correlations of 72 automatic metrics with human criteria. Our analysis highlights the weaknesses of current metrics for ASG and allows us to formulate practical recommendations for ASG evaluation.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Builds on 60 of its 132 references that are in the map. First posted on arXiv 24 Aug 2022.

Builds on

60 of 132 references.

References · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2018Hierarchical Neural Story Generationcited by 1,193
  10. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  11. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  12. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
38 more
  1. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  2. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  3. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  4. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 294
  5. 2018Plan-And-Write: Towards Better Automatic Storytellingcited by 234
  6. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  7. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  8. 2016Visual Storytellingcited by 176
  9. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 158
  10. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  11. 2019Handling Divergent Reference Texts when Evaluating Table-to-Text Generationcited by 119
  12. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116
  13. 2020PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Trackingcited by 105
  14. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 97
  15. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  16. 2020Content Planning for Neural Story Generation with Aristotelian Rescoringcited by 92
  17. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 82
  18. 2016Argumentation Mining in User-Generated Web Discoursecited by 69
  19. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  20. 2015Better Summarization Evaluation with Word Embeddings for ROUGEcited by 66
  21. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  22. 2018Story Ending Generation with Incremental Encoding and Commonsense Knowledgecited by 60
  23. 2021A Plug-and-Play Method for Controlled Text Generationcited by 56
  24. 2018Towards a Better Metric for Evaluating Question Generation Systemscited by 52
  25. 2019Affect-Driven Dialog Generationcited by 50
  26. 2021Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherencecited by 45
  27. 2021Transformer-based Conditional Variational Autoencoder for Controllable Story Generationcited by 44
  28. 2018A Skeleton-Based Model for Promoting Coherence Among Sentences in Narrative Story Generationcited by 42
  29. 2020Modeling Protagonist Emotions for Emotion-Aware Storytellingcited by 42
  30. 2019Crowdsourcing Lightweight Pyramids for Manual Summary Evaluationcited by 39
  31. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  32. 2021A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representationscited by 37
  33. 2020Hierarchical Pre-training for Sequence Labelling in Spoken Dialogcited by 36
  34. 2021InfoLM: A New Metric to Evaluate Summarization & Data2Text Generationcited by 33
  35. 2020Bringing Stories Alive: Generating Interactive Fiction Worldscited by 28
  36. 2021Automatic Story Generation: Challenges and Attemptscited by 28
  37. 2020Guiding attention in Sequence-to-sequence models for Dialogue Act predictioncited by 26
  38. 2020Heavy-tailed Representations, Text Polarity Classification & Data Augmentationcited by 20

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 14; 2024: 14; 2025: 19; 2026: 25.

75 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  3. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  4. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  5. 2023The Next Chapter: A Study of Large Language Models in Storytellingcited by 58
  6. 2024Agents' Room: Narrative Generation through Multi-step Collaborationcited by 57
  7. 2025SCORE: Story Coherence and Retrieval Enhancement for AI Narrativescited by 42
  8. 2024Benchmarking Linguistic Diversity of Large Language Modelscited by 38
  9. 2024Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluationcited by 32
  10. 2023Learning Personalized Alignment for Evaluating Open-ended Text Generationcited by 29
  11. 2023A Family of Pretrained Transformer Language Models for Russiancited by 27
  12. 2024MERA: A Comprehensive LLM Evaluation in Russiancited by 22
38 more
  1. 2024Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisonscited by 22
  2. 2024Evaluating Creative Short Story Generation in Humans and Large Language Modelscited by 21
  3. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  4. 2025StoryWriter: A Multi-Agent Framework for Long Story Generationcited by 19
  5. 2023BatchEval: Towards Human-like Text Evaluationcited by 17
  6. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  7. 2024Bayesian Calibration of Win Rate Estimation with LLM Evaluatorscited by 16
  8. 2024Navigating the Path of Writing: Outline-guided Text Generation with Large Language Modelscited by 15
  9. 2023Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarkscited by 14
  10. 2026Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluationcited by 13
  11. 2023Affective and Dynamic Beam Search for Story Generationcited by 12
  12. 2023LongStory: Coherent, Complete and Length Controlled Long story Generationcited by 12
  13. 2023Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applicationscited by 11
  14. 2023Event Causality Is Key to Computational Story Understandingcited by 11
  15. 2025Once Upon a Time: Interactive Learning for Storytelling with Small Language Modelscited by 11
  16. 2022The Glass Ceiling of Automatic Evaluation in Natural Language Generationcited by 10
  17. 2023DeltaScore: Fine-Grained Story Evaluation with Perturbationscited by 9
  18. 2025The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writingcited by 8
  19. 2026Who can we trust? LLM-as-a-jury for Comparative Assessmentcited by 7
  20. 2023STORYWARS: A Dataset and Instruction Tuning Baselines for Collaborative Story Understanding and Generationcited by 5
  21. 2023HAUSER: Towards Holistic and Automatic Evaluation of Simile Generationcited by 5
  22. 2025HPSS: Heuristic Prompting Strategy Search for LLM Evaluatorscited by 5
  23. 2025What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluationcited by 5
  24. 2026Narrative Theory-Driven LLM Methods for Automatic Story Generation and Understanding: A Surveycited by 5
  25. 2024Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluationcited by 4
  26. 2025OpeNLGauge: An Explainable Metric for NLG Evaluation with Open-Weights LLMscited by 4
  27. 2025Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judgecited by 4
  28. 2025FedRAG: A Framework for Fine-Tuning Retrieval-Augmented Generation Systemscited by 4
  29. 2022Beyond Mahalanobis-Based Scores for Textual OOD Detectioncited by 3
  30. 2025Re:Verse -- Can Your VLM Read a Manga?cited by 3
  31. 2025Large Language Models as Search Engines: Societal Challengescited by 3
  32. 2022Little Red Riding Hood Goes Around the Globe:Crosslingual Story Planning and Generation with Large Language Modelscited by 2
  33. 2025From Sub-Ability Diagnosis to Human-Aligned Generation: Bridging the Gap for Text Length Control via MARKERGENcited by 2
  34. 2025Can LLMs Generate Good Stories? Insights and Challenges from a Narrative Planning Perspectivecited by 2
  35. 2025Plug-and-Play Dramaturge: A Divide-and-Conquer Approach for Iterative Narrative Script Refinement via Collaborative LLM Agentscited by 2
  36. 2026Writer-R1: Enhancing Generative Writing in LLMs via Memory-augmented Replay Policy Optimizationcited by 2
  37. 2026Who Wrote the Book? Detecting and Attributing LLM Ghostwriterscited by 2
  38. 2026DramaDirector: Geometry-Guided Short Drama Generationcited by 2

50 most cited of 75 All 75 (JSON)