# Plot-guided Adversarial Example Construction for Evaluating Open-domain Story Generation arXiv: https://arxiv.org/abs/2104.05801 ยท HTML: https://mlsanity.com/paper/2104.05801/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 33 references that are in the map. - First posted on arXiv 12 Apr 2021. ## Builds on (most cited first, 22 of 22) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020) - [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020) - [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020) - [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018) - [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019) - [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019) - [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018) - [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018) - [Strategies for Structuring Story Generation](https://mlsanity.com/paper/1902.01109/) (2019) - [Learning to Write with Cooperative Discriminators](https://mlsanity.com/paper/1805.06087/) (2018) - [PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking](https://mlsanity.com/paper/2004.14967/) (2020) - [Content Planning for Neural Story Generation with Aristotelian Rescoring](https://mlsanity.com/paper/2009.09870/) (2020) - [UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation](https://mlsanity.com/paper/2009.07602/) (2020) - [Plan, Write, and Revise: an Interactive System for Open-Domain Story Generation](https://mlsanity.com/paper/1904.02357/) (2019) - [Neural Net Models for Open-Domain Discourse Coherence](https://mlsanity.com/paper/1606.01545/) (2016) - [Discourse Coherence in the Wild: A Dataset, Evaluation and Methods](https://mlsanity.com/paper/1805.04993/) (2018) - [Cue Me In: Content-Inducing Approaches to Interactive Story Generation](https://mlsanity.com/paper/2010.09935/) (2020) ## Built on by (most cited first, 13 of 13) - [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022) - [DialFact: A Benchmark for Fact-Checking in Dialogue](https://mlsanity.com/paper/2110.08222/) (2021) - [Learning Personalized Alignment for Evaluating Open-ended Text Generation](https://mlsanity.com/paper/2310.03304/) (2023) - [DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations](https://mlsanity.com/paper/2203.09711/) (2022) - [StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoning](https://mlsanity.com/paper/2210.08459/) (2022) - [EtriCA: Event-Triggered Context-Aware Story Generation Augmented by Cross Attention](https://mlsanity.com/paper/2210.12463/) (2022) - [What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation](https://mlsanity.com/paper/2408.14622/) (2024) - [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022) - [DeltaScore: Fine-Grained Story Evaluation with Perturbations](https://mlsanity.com/paper/2303.08991/) (2023) - [A Cross-Attention Augmented Model for Event-Triggered Context-Aware Story Generation](https://mlsanity.com/paper/2311.11271/) (2023) - [What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluation](https://mlsanity.com/paper/2512.12839/) (2025) - [Explaining Relationships Among Research Papers](https://mlsanity.com/paper/2402.13426/) (2024) - [CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization](https://mlsanity.com/paper/2503.17136/) (2025) ## Abstract (arXiv) With the recent advances of open-domain story generation, the lack of reliable automatic evaluation metrics becomes an increasingly imperative issue that hinders the fast development of story generation. According to conducted researches in this regard, learnable evaluation metrics have promised more accurate assessments by having higher correlations with human judgments. A critical bottleneck of obtaining a reliable learnable evaluation metric is the lack of high-quality training data for classifiers to efficiently distinguish plausible and implausible machine-generated stories. Previous works relied on textitheuristically manipulated plausible examples to mimic possible system drawbacks such as repetition, contradiction, or irrelevant content in the text level, which can be textitunnatural and textitoversimplify the characteristics of implausible machine-generated stories. We propose to tackle these issues by generating a more comprehensive set of implausible stories using em plots, which are structured representations of controllable factors used to generate stories. Since these plots are compact and structured, it is easier to manipulate them to generate text with targeted undesirable properties, while at the same time maintain the grammatical correctness and naturalness of the generated sentences. To improve the quality of generated implausible stories, we further apply the adversarial filtering procedure presented by citetzellers2018swag to select a more nuanced set of implausible texts. Experiments show that the evaluation metrics trained on our generated data result in more reliable automatic assessments that correlate remarkably better with human judgments compared to the baselines.