# What do Large Language Models Learn about Scripts? arXiv: https://arxiv.org/abs/2112.13834 ยท HTML: https://mlsanity.com/paper/2112.13834/ ## Key facts - Cited by 19 papers in the map (data through 14 Oct 2026). - Builds on 20 of its 45 references that are in the map. - First posted on arXiv 27 Dec 2021. ## Builds on (most cited first, 20 of 20) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019) - [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019) - [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016) - [Commonsense Knowledge Mining from Pretrained Models](https://mlsanity.com/paper/1909.00505/) (2019) - [How Context Affects Language Models' Factual Predictions](https://mlsanity.com/paper/2005.04611/) (2020) - [Unsupervised Commonsense Question Answering with Self-Talk](https://mlsanity.com/paper/2004.05483/) (2020) - [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020) - [Inducing Relational Knowledge from BERT](https://mlsanity.com/paper/1911.12753/) (2019) - [Reasoning about Goals, Steps, and Temporal Ordering with WikiHow](https://mlsanity.com/paper/2009.07690/) (2020) - [Temporal Common Sense Acquisition with Minimal Supervision](https://mlsanity.com/paper/2005.04304/) (2020) - [MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge](https://mlsanity.com/paper/1803.05223/) (2018) - [Goal-Oriented Script Construction](https://mlsanity.com/paper/2107.13189/) (2021) - [Analogous Process Structure Induction for Sub-event Sequence Prediction](https://mlsanity.com/paper/2010.08525/) (2020) - [proScript: Partially Ordered Scripts Generation via Pre-trained Language Models](https://mlsanity.com/paper/2104.08251/) (2021) - [MCScript2.0: A Machine Comprehension Corpus Focused on Script Events and Participants](https://mlsanity.com/paper/1905.09531/) (2019) - [Learning Household Task Knowledge from WikiHow Descriptions](https://mlsanity.com/paper/1909.06414/) (2019) - [Using Sentence-Level LSTM Language Models for Script Inference](https://mlsanity.com/paper/1604.02993/) (2016) - [InScript: Narrative texts annotated with script information](https://mlsanity.com/paper/1703.05260/) (2017) ## Built on by (most cited first, 19 of 19) - [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023) - [Distilling Script Knowledge from Large Language Models for Constrained Language Planning](https://mlsanity.com/paper/2305.05252/) (2023) - [Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?](https://mlsanity.com/paper/2406.10486/) (2024) - [Natural Language Decompositions of Implicit Content Enable Better Text Representations](https://mlsanity.com/paper/2305.14583/) (2023) - [Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification](https://mlsanity.com/paper/2307.01972/) (2023) - [Multimedia Generative Script Learning for Task Planning](https://mlsanity.com/paper/2208.12306/) (2022) - [PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilities](https://mlsanity.com/paper/2504.14773/) (2025) - [Unsupervised Task Graph Generation from Instructional Video Transcripts](https://mlsanity.com/paper/2302.09173/) (2023) - [Dynamic Large Language Models on Blockchains](https://mlsanity.com/paper/2307.10549/) (2023) - [Multilevel Large Language Models for Everyone](https://mlsanity.com/paper/2307.13221/) (2023) - [PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) Planning](https://mlsanity.com/paper/2305.19472/) (2023) - [Gradient Domain Diffusion Models for Image Synthesis](https://mlsanity.com/paper/2309.01875/) (2023) - [Do large language models and humans have similar behaviors in causal inference with script knowledge?](https://mlsanity.com/paper/2311.07311/) (2023) - [ScriptWorld: Text Based Environment For Learning Procedural Knowledge](https://mlsanity.com/paper/2307.03906/) (2023) - [Formulation Comparison for Timeline Construction using LLMs](https://mlsanity.com/paper/2403.00990/) (2024) - [TSSR: A Truncated and Signed Square Root Activation Function for Neural Networks](https://mlsanity.com/paper/2308.04832/) (2023) - [ScEdit: Script-based Assessment of Knowledge Editing](https://mlsanity.com/paper/2505.23291/) (2025) - [STL: A Signed and Truncated Logarithm Activation Function for Neural Networks](https://mlsanity.com/paper/2307.16389/) (2023) - [PLMM: Personal Large Language Models on Mobile Devices](https://mlsanity.com/paper/2309.14726/) (2023) ## Abstract (arXiv) Script Knowledge (Schank and Abelson, 1975) has long been recognized as crucial for language understanding as it can help in filling in unstated information in a narrative. However, such knowledge is expensive to produce manually and difficult to induce from text due to reporting bias (Gordon and Van Durme, 2013). In this work, we are interested in the scientific question of whether explicit script knowledge is present and accessible through pre-trained generative language models (LMs). To this end, we introduce the task of generating full event sequence descriptions (ESDs) given a scenario in the form of natural language prompts. In zero-shot probing experiments, we find that generative LMs produce poor ESDs with mostly omitted, irrelevant, repeated or misordered events. To address this, we propose a pipeline-based script induction framework (SIF) which can generate good quality ESDs for unseen scenarios (e.g., bake a cake). SIF is a two-staged framework that fine-tunes LM on a small set of ESD examples in the first stage. In the second stage, ESD generated for an unseen scenario is post-processed using RoBERTa-based models to filter irrelevant events, remove repetitions, and reorder the temporally misordered events. Through automatic and manual evaluations, we demonstrate that SIF yields substantial improvements (1-3 BLUE points) over a fine-tuned LM. However, manual analysis shows that there is great room for improvement, offering a new research direction for inducing script knowledge.