mlsanityData to 14 Oct 2026

Cut the CARP: Fishing for zero-shot story evaluation

arXiv 2110.03111 ↗First posted 6 Oct 2021cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Recent advances in large-scale language models (Raffel et al., 2019; Brown et al., 2020) have brought significant qualitative and quantitative improvements in machine-driven text generation. Despite this, generation and evaluation of machine-generated narrative text remains a challenging problem. Objective evaluation of computationally-generated stories may be prohibitively expensive, require meticulously annotated datasets, or may not adequately measure the logical coherence of a generated story's narratological structure. Informed by recent advances in contrastive learning (Radford et al., 2021), we present Contrastive Authoring and Reviewing Pairing (CARP): a scalable, efficient method for performing qualitatively superior, zero-shot evaluation of stories. We show a strong correlation between human evaluation of stories and those of CARP. Model outputs more significantly correlate with corresponding human input than those language-model based methods which utilize finetuning or prompt engineering approaches. We also present and analyze the Story-Critique Dataset, a new corpora composed of 1.3 million aligned story-critique pairs derived from over 80,000 stories. We expect this corpus to be of interest to NLP researchers.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 30 of its 48 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

30 of 48 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2018Hierarchical Neural Story Generationcited by 1,193
  10. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  12. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
18 more
  1. 2018Plan-And-Write: Towards Better Automatic Storytellingcited by 234
  2. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 160
  3. 2019Do Massively Pretrained Language Models Make Better Storytellers?cited by 90
  4. 2020MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Modelscited by 74
  5. 2021Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluationcited by 70
  6. 2019Question Answering as an Automatic Evaluation Metric for News Article Summarizationcited by 63
  7. 2021Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrievalcited by 63
  8. 2020STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generationcited by 56
  9. 2017QMDP-Net: Deep Learning for Planning under Partial Observabilitycited by 51
  10. 2021Transformer-based Conditional Variational Autoencoder for Controllable Story Generationcited by 44
  11. 2016Revisiting Summarization Evaluation for Scientific Articlescited by 31
  12. 2021Inferring the Reader: Guiding Automated Story Generation with Commonsense Reasoningcited by 29
  13. 2020Collaborative Storytelling with Large-scale Neural Language Modelscited by 21
  14. 2020Pitfalls in Machine Learning Research: Reexamining the Development Cyclecited by 17
  15. 2021AffectGAN: Affect-Based Generative Art Driven by Semanticscited by 10
  16. 2021Fabula Entropy Indexing: Objective Measures of Story Coherencecited by 7
  17. 2021Towards a Formal Model of Narrativescited by 6
  18. 2021Zero-Shot Controlled Generation with Encoder-Decoder Transformerscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 4; 2024: 5.

14 papers.

Sort:

By area:

  1. 2022Self-critiquing models for assisting human evaluatorscited by 380
  2. 2023GPTScore: Evaluate as You Desirecited by 361
  3. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 165
  4. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  5. 2023Training Language Models with Language Feedback at Scalecited by 128
  6. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoningcited by 88
  7. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  8. 2022Training Language Models with Language Feedbackcited by 49
  9. 2022Datasheet for the Pilecited by 43
  10. 2023Beyond Prompts: Exploring the Design Space of Mixed-Initiative Co-Creativity Systemscited by 20
  11. 2024Suppressing Pink Elephants with Direct Principle Feedbackcited by 15
  12. 2022Robust Preference Learning for Storytelling via Contrastive Reinforcement Learningcited by 14
2 more
  1. 2024Improving Black-box Robustness with In-Context Rewritingcited by 8
  2. 2024Topological Perspectives on Optimal Multimodal Embedding Spacescited by 3