mlsanityData to 13 Oct 2026

Do Massively Pretrained Language Models Make Better Storytellers?

arXiv 1909.10705 ↗First posted 24 Sep 2019cs.CL cs.AI cs.LGcited by 90

Not on the map. Why?

Abstract (arXiv)

Large neural language models trained on massive amounts of text have emerged as a formidable strategy for Natural Language Understanding tasks. However, the strength of these models as Natural Language Generators is less clear. Though anecdotal evidence suggests that these models generate better quality text, there has been no detailed study characterizing their generation abilities. In this work, we compare the performance of an extensively pretrained model, OpenAI GPT2-117 (Radford et al., 2019), to a state-of-the-art neural story generation model (Fan et al., 2018). By evaluating the generated text across a wide variety of automatic metrics, we characterize the ways in which pretrained models do, and do not, make better storytellers. We find that although GPT2-117 conditions more strongly on context, is more sensitive to ordering of events, and uses more unusual words, it is just as likely to produce repetitive and under-diverse text when using likelihood-maximizing decoding algorithms.

Key facts. Cited by 90 papers in the map (data through 13 Oct 2026). First posted on arXiv 24 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 25; 2022: 25; 2023: 20; 2024: 10; 2025: 6; 2026: 4.

90 papers.

Sort:

By area:

  1. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 423
  2. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  3. 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
  4. 2023Does Writing with Language Models Reduce Content Diversity?cited by 191
  5. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 137
  6. 2024Homogenization Effects of Large Language Models on Human Creative Ideationcited by 127
  7. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  8. 2021TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generationcited by 119
  9. 2022Locally Typical Samplingcited by 109
  10. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 98
  11. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  12. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
38 more
  1. 2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 79
  2. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  3. 2022Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writingcited by 71
  4. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  5. 2023The Next Chapter: A Study of Large Language Models in Storytellingcited by 58
  6. 2021Transformer-based Conditional Variational Autoencoder for Controllable Story Generationcited by 44
  7. 2021Generate, Prune, Select: A Pipeline for Counterspeech Generation against Online Hate Speechcited by 42
  8. 2023Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Modelscited by 37
  9. 2022Dungeons and Dragons as a Dialog Challenge for Artificial Intelligencecited by 35
  10. 2024IBSEN: Director-Actor Agent Collaboration for Controllable and Interactive Drama Script Generationcited by 35
  11. 2024Suri: Multi-constraint Instruction Following for Long-form Text Generationcited by 34
  12. 2022Unified Detoxifying and Debiasing in Language Generation via Inference-time Adaptive Optimizationcited by 33
  13. 2022CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Data Limitation With Contrastive Learningcited by 32
  14. 2022Visualize Before You Write: Imagination-Guided Open-Ended Text Generationcited by 30
  15. 2022How Far are We from Robust Long Abstractive Summarization?cited by 25
  16. 2022Open-world Story Generation with Structured Knowledge Enhancement: A Comprehensive Surveycited by 22
  17. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 20
  18. 2021Relating Neural Text Degeneration to Exposure Biascited by 19
  19. 2023Stealing the Decoding Algorithms of Language Modelscited by 19
  20. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
  21. 2025Unlocking Efficient, Scalable, and Continual Knowledge Editing with Basis-Level Representation Fine-Tuningcited by 18
  22. 2024Small Language Models can Outperform Humans in Short Creative Writing: A Study Comparing SLMs with Humans and LLMscited by 17
  23. 2023Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Modelscited by 16
  24. 2021Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Modelscited by 15
  25. 2021NAREOR: The Narrative Reordering Problemcited by 14
  26. 2021Inspiration through Observation: Demonstrating the Influence of Automatically Generated Text on Creative Writingcited by 14
  27. 2021Cut the CARP: Fishing for zero-shot story evaluationcited by 14
  28. 2021DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformercited by 14
  29. 2021Goal-Directed Story Generation: Augmenting Generative Language Models with Reinforcement Learningcited by 14
  30. 2021Coarse2Fine: Fine-grained Text Classification on Coarsely-grained Annotated Datacited by 13
  31. 2023Storyfier: Exploring Vocabulary Learning Support with Text Generation Modelscited by 12
  32. 2023How Far Are LLMs from Believable AI? A Benchmark for Evaluating the Believability of Human Behavior Simulationcited by 11
  33. 2025Once Upon a Time: Interactive Learning for Storytelling with Small Language Modelscited by 11
  34. 2022Innovations in Neural Data-to-text Generation: A Surveycited by 10
  35. 2021Outline to Story: Fine-grained Controllable Story Generation from Cascaded Eventscited by 8
  36. 2022What Makes Data-to-Text Generation Hard for Pretrained Language Models?cited by 8
  37. 2021Changing the Mind of Transformers for Topically-Controllable Language Generationcited by 7
  38. 2022CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understandingcited by 7

50 most cited of 90 All 90 (JSON)