# proScript: Partially Ordered Scripts Generation via Pre-trained Language Models

arXiv: https://arxiv.org/abs/2104.08251 · HTML: https://mlsanity.com/paper/2104.08251/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking](https://mlsanity.com/paper/2004.14967/) (2020)
- [Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension](https://mlsanity.com/paper/1805.06975/) (2018)
- [Simulating Action Dynamics with Neural Process Networks](https://mlsanity.com/paper/1711.05313/) (2017)
- [Tracking the World State with Recurrent Entity Networks](https://mlsanity.com/paper/1612.03969/) (2016)
- [A Dataset for Tracking Entities in Open Domain Procedural Text](https://mlsanity.com/paper/2011.08092/) (2020)
- [Neural Language Modeling for Contextualized Temporal Graph Generation](https://mlsanity.com/paper/2010.10077/) (2020)
- [Hierarchical Quantized Representations for Script Generation](https://mlsanity.com/paper/1808.09542/) (2018)
- [InScript: Narrative texts annotated with script information](https://mlsanity.com/paper/1703.05260/) (2017)
- [Modeling Semantic Expectation: Using Script Knowledge for Referent Prediction](https://mlsanity.com/paper/1702.03121/) (2017)
- [Machine-Assisted Script Curation](https://mlsanity.com/paper/2101.05400/) (2021)

## Built on by (most cited first, 21 of 21)

- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models](https://mlsanity.com/paper/2309.10062/) (2023)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Distilling Script Knowledge from Large Language Models for Constrained Language Planning](https://mlsanity.com/paper/2305.05252/) (2023)
- [Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedback](https://mlsanity.com/paper/2112.09737/) (2021)
- [JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents](https://mlsanity.com/paper/2208.13266/) (2022)
- [Pretrained Language Models as Visual Planners for Human Assistance](https://mlsanity.com/paper/2304.09179/) (2023)
- [I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation](https://mlsanity.com/paper/2212.09246/) (2022)
- [Think about it! Improving defeasible reasoning by first modeling the question scenario](https://mlsanity.com/paper/2110.12349/) (2021)
- [What do Large Language Models Learn about Scripts?](https://mlsanity.com/paper/2112.13834/) (2021)
- [Can LLMs Reason in the Wild with Programs?](https://mlsanity.com/paper/2406.13764/) (2024)
- [TaskLAMA: Probing the Complex Task Understanding of Language Models](https://mlsanity.com/paper/2308.15299/) (2023)
- [LongStory: Coherent, Complete and Length Controlled Long story Generation](https://mlsanity.com/paper/2311.15208/) (2023)
- [Interscript: A dataset for interactive learning of scripts through error feedback](https://mlsanity.com/paper/2112.07867/) (2021)
- [Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning](https://mlsanity.com/paper/2204.04813/) (2022)
- [Infinite Ends from Finite Samples: Open-Ended Goal Inference as Top-Down Bayesian Filtering of Bottom-Up Proposals](https://mlsanity.com/paper/2407.16770/) (2024)
- [The Compressor-Retriever Architecture for Language Model OS](https://mlsanity.com/paper/2409.01495/) (2024)
- [Towards Quantifying Commonsense Reasoning with Mechanistic Insights](https://mlsanity.com/paper/2504.10077/) (2025)
- [RAG-based Crowdsourcing Task Decomposition via Masked Contrastive Learning with Prompts](https://mlsanity.com/paper/2406.06577/) (2024)
- [Large Language Models as Common-Sense Heuristics](https://mlsanity.com/paper/2501.18816/) (2025)

## Abstract (arXiv)

Scripts - standardized event sequences describing typical everyday activities - have been shown to help understand narratives by providing expectations, resolving ambiguity, and filling in unstated information. However, to date they have proved hard to author or extract from text. In this work, we demonstrate for the first time that pre-trained neural language models (LMs) can be be finetuned to generate high-quality scripts, at varying levels of granularity, for a wide range of everyday scenarios (e.g., bake a cake). To do this, we collected a large (6.4k), crowdsourced partially ordered scripts (named proScript), which is substantially larger than prior datasets, and developed models that generate scripts with combining language generation and structure prediction. We define two complementary tasks: (i) edge prediction: given a scenario and unordered events, organize the events into a valid (possibly partial-order) script, and (ii) script generation: given only a scenario, generate events and organize them into a (possibly partial-order) script. Our experiments show that our models perform well (e.g., F1=75.7 in task (i)), illustrating a new approach to overcoming previous barriers to script collection. We also show that there is still significant room for improvement toward human level performance. Together, our tasks, dataset, and models offer a new research direction for learning script knowledge.