# A Dataset for Tracking Entities in Open Domain Procedural Text

arXiv: https://arxiv.org/abs/2011.08092 · HTML: https://mlsanity.com/paper/2011.08092/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2020.

## Built on by (most cited first, 30 of 37)

- [Distilling Script Knowledge from Large Language Models for Constrained Language Planning](https://mlsanity.com/paper/2305.05252/) (2023)
- [SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.01599/) (2024)
- [Causal Reasoning of Entities and Events in Procedural Texts](https://mlsanity.com/paper/2301.10896/) (2023)
- [proScript: Partially Ordered Scripts Generation via Pre-trained Language Models](https://mlsanity.com/paper/2104.08251/) (2021)
- [On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations](https://mlsanity.com/paper/2204.12708/) (2022)
- [Understanding Multimodal Procedural Knowledge by Sequencing Multimodal Instructional Manuals](https://mlsanity.com/paper/2110.08486/) (2021)
- [Process-Level Representation of Scientific Protocols with Interactive Annotation](https://mlsanity.com/paper/2101.10244/) (2021)
- [Multimedia Generative Script Learning for Task Planning](https://mlsanity.com/paper/2208.12306/) (2022)
- [CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans](https://mlsanity.com/paper/2406.15823/) (2024)
- [Coalescing Global and Local Information for Procedural Text Understanding](https://mlsanity.com/paper/2208.12848/) (2022)
- [Understand the Dynamic World: An End-to-End Knowledge Informed Framework for Open Domain Entity State Tracking](https://mlsanity.com/paper/2304.13854/) (2023)
- [Reasoning about Procedures with Natural Language Processing: A Tutorial](https://mlsanity.com/paper/2205.07455/) (2022)
- [Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows](https://mlsanity.com/paper/2209.05840/) (2022)
- [OpenPI2.0: An Improved Dataset for Entity Tracking in Texts](https://mlsanity.com/paper/2305.14603/) (2023)
- [Localizing Active Objects from Egocentric Vision with Symbolic World Knowledge](https://mlsanity.com/paper/2310.15066/) (2023)
- [Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarking](https://mlsanity.com/paper/2112.00086/) (2021)
- [EvEntS ReaLM: Event Reasoning of Entity States via Language Models](https://mlsanity.com/paper/2211.05392/) (2022)
- [Exploring the Curious Case of Code Prompts](https://mlsanity.com/paper/2304.13250/) (2023)
- [EnigmaToM: Improve LLMs' Theory-of-Mind Reasoning Capabilities with Neural Knowledge Base of Entity States](https://mlsanity.com/paper/2503.03340/) (2025)
- [LEMON: Language-Based Environment Manipulation via Execution-Guided Pre-training](https://mlsanity.com/paper/2201.08081/) (2022)
- [PASTA: A Dataset for Modeling Participant States in Narratives](https://mlsanity.com/paper/2208.00329/) (2022)
- [SOCCER: An Information-Sparse Discourse State Tracking Collection in the Sports Commentary Domain](https://mlsanity.com/paper/2106.01972/) (2021)
- [Learning Action Conditions from Instructional Manuals for Instruction Understanding](https://mlsanity.com/paper/2205.12420/) (2022)
- [The Role of Semantic Parsing in Understanding Procedural Text](https://mlsanity.com/paper/2302.06829/) (2023)
- [PAGED: A Benchmark for Procedural Graphs Extraction from Documents](https://mlsanity.com/paper/2408.03630/) (2024)
- [Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension](https://mlsanity.com/paper/2204.02566/) (2022)
- [Choice-75: A Dataset on Decision Branching in Script Learning](https://mlsanity.com/paper/2309.11737/) (2023)
- [A Neural Edge-Editing Approach for Document-Level Relation Graph Extraction](https://mlsanity.com/paper/2106.09900/) (2021)
- [Dense Paraphrasing for Textual Enrichment](https://mlsanity.com/paper/2210.11563/) (2022)
- [OpenPI-C: A Better Benchmark and Stronger Baseline for Open-Vocabulary State Tracking](https://mlsanity.com/paper/2306.00887/) (2023)

## Abstract (arXiv)

We present the first dataset for tracking state changes in procedural text from arbitrary domains by using an unrestricted (open) vocabulary. For example, in a text describing fog removal using potatoes, a car window may transition between being foggy, sticky,opaque, and clear. Previous formulations of this task provide the text and entities involved,and ask how those entities change for just a small, pre-defined set of attributes (e.g., location), limiting their fidelity. Our solution is a new task formulation where given just a procedural text as input, the task is to generate a set of state change tuples(entity, at-tribute, before-state, after-state)for each step,where the entity, attribute, and state values must be predicted from an open vocabulary. Using crowdsourcing, we create OPENPI1, a high-quality (91.5% coverage as judged by humans and completely vetted), and large-scale dataset comprising 29,928 state changes over 4,050 sentences from 810 procedural real-world paragraphs from WikiHow.com. A current state-of-the-art generation model on this task achieves 16.1% F1 based on BLEU metric, leaving enough room for novel model architectures.