mlsanityData to 14 Oct 2026

Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation

arXiv 2405.13769 ↗First posted 22 May 2024cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Storytelling is an integral part of human experience and plays a crucial role in social interactions. Thus, Automatic Story Evaluation (ASE) and Generation (ASG) could benefit society in multiple ways, but they are challenging tasks which require high-level human abilities such as creativity, reasoning and deep understanding. Meanwhile, Large Language Models (LLM) now achieve state-of-the-art performance on many NLP tasks. In this paper, we study whether LLMs can be used as substitutes for human annotators for ASE. We perform an extensive analysis of the correlations between LLM ratings, other automatic measures, and human annotations, and we explore the influence of prompting on the results and the explainability of LLM behaviour. Most notably, we find that LLMs outperform current automatic measures for system-level evaluation but still struggle at providing satisfactory explanations for their answers.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 48 of its 108 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

48 of 108 references.

References · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
36 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  5. 2018Hierarchical Neural Story Generationcited by 1,193
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  7. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  8. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  9. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  10. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  11. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  12. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  13. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  14. 2020Evaluation of Text Generation: A Surveycited by 277
  15. 2022Data Contamination: From Memorization to Exploitationcited by 236
  16. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  17. 2022Is GPT-3 a Good Data Annotator?cited by 208
  18. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  19. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  20. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 172
  21. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  22. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  23. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116
  24. 2020PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Trackingcited by 105
  25. 2020Content Planning for Neural Story Generation with Aristotelian Rescoringcited by 92
  26. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  27. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 82
  28. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 75
  29. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  30. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  31. 2021Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherencecited by 45
  32. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  33. 2021Automatic Story Generation: Challenges and Attemptscited by 28
  34. 2022The Glass Ceiling of Automatic Evaluation in Natural Language Generationcited by 10
  35. 2017Abandon Statistical Significancecited by 5
  36. 2021A Temporal Variational Model for Story Generationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 11; 2026: 15.

32 papers.

Sort:

By area:

  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  2. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 59
  3. 2024Agents' Room: Narrative Generation through Multi-step Collaborationcited by 57
  4. 2025SCORE: Story Coherence and Retrieval Enhancement for AI Narrativescited by 42
  5. 2024Evaluating Creative Short Story Generation in Humans and Large Language Modelscited by 21
  6. 2025StoryWriter: A Multi-Agent Framework for Long Story Generationcited by 19
  7. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  8. 2026Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluationcited by 13
  9. 2026Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agentscited by 10
  10. 2024EmoDynamiX: Emotional Support Dialogue Strategy Prediction by Modelling MiXed Emotions and Discourse Dynamicscited by 9
  11. 2025Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probescited by 9
  12. 2025The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writingcited by 8
20 more
  1. 2024Creativity in AI: Progresses and Challengescited by 7
  2. 2026Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judgecited by 6
  3. 2025What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluationcited by 5
  4. 2026Narrative Theory-Driven LLM Methods for Automatic Story Generation and Understanding: A Surveycited by 5
  5. 2026Rewarding Creativity: A Human-Aligned Generative Reward Model for Reinforcement Learning in Storytellingcited by 2
  6. 2026Automated Creativity Evaluation of Language Models Across Open-Ended Taskscited by 2
  7. 2025A Dual-Perspective NLG Meta-Evaluation Framework with Automatic Benchmark and Better Interpretabilitycited by 1
  8. 2026StoryAlign: Evaluating and Training Reward Models for Story Generationcited by 1
  9. 2026Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliabilitycited by 1
  10. 2025The Anatomy of Speech Persuasion: Linguistic Shifts in LLM-Modified Speechescited by 0
  11. 2025From Image Captioning to Visual Storytellingcited by 0
  12. 2025Direct-Scoring NLG Evaluators Can Use Pairwise Comparisons Toocited by 0
  13. 2025Applying Large Language Models to Characterize Public Narrativescited by 0
  14. 2026LLMs Should Incorporate Explicit Mechanisms for Human Empathycited by 0
  15. 2026Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Frameworkcited by 0
  16. 2026Steering Emotional Dynamics for Art Therapy: Controllable Narrative Script Generation through Hierarchically Guided LLM Agentscited by 0
  17. 2026Identifying Implicit Bias in LLM-based Chat AI Toward People with Intellectual Disabilitiescited by 0
  18. 2026Inter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended Textcited by 0
  19. 2026The Effects of Incremental Instruction Delivery on Language-Model Creative Writingcited by 0
  20. 2026Scaling Long-Form Story Generation via Narrative State Trackingcited by 0