# Knowledge-Aware Procedural Text Understanding with Multi-Stage Training

arXiv: https://arxiv.org/abs/2009.13199 · HTML: https://mlsanity.com/paper/2009.13199/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2020.

## Built on by (most cited first, 17 of 17)

- [A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods](https://mlsanity.com/paper/2204.03508/) (2022)
- [Large Language Models are Built-in Autoregressive Search Engines](https://mlsanity.com/paper/2305.09612/) (2023)
- [Time-Stamped Language Model: Teaching Language Models to Understand the Flow of Events](https://mlsanity.com/paper/2104.07635/) (2021)
- [Coalescing Global and Local Information for Procedural Text Understanding](https://mlsanity.com/paper/2208.12848/) (2022)
- [A Unified Encoder-Decoder Framework with Entity Memory](https://mlsanity.com/paper/2210.03273/) (2022)
- [Pre-training Language Models for Comparative Reasoning](https://mlsanity.com/paper/2305.14457/) (2023)
- [LEMON: Language-Based Environment Manipulation via Execution-Guided Pre-training](https://mlsanity.com/paper/2201.08081/) (2022)
- [The Role of Semantic Parsing in Understanding Procedural Text](https://mlsanity.com/paper/2302.06829/) (2023)
- [Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension](https://mlsanity.com/paper/2204.02566/) (2022)
- [Adversarial Transformer Language Models for Contextual Commonsense Inference](https://mlsanity.com/paper/2302.05406/) (2023)
- [Transferring Procedural Knowledge across Commonsense Tasks](https://mlsanity.com/paper/2304.13867/) (2023)
- [Order-Based Pre-training Strategies for Procedural Text Understanding](https://mlsanity.com/paper/2404.04676/) (2024)
- [Procedural Text Understanding via Scene-Wise Evolution](https://mlsanity.com/paper/2203.07600/) (2022)
- [Entity Tracking via Effective Use of Multi-Task Learning Model and Mention-guided Decoding](https://mlsanity.com/paper/2210.06444/) (2022)
- [Can Language Models Take A Hint? Prompting for Controllable Contextualized Commonsense Inference](https://mlsanity.com/paper/2410.02202/) (2024)
- [Knowing-how & Knowing-that: A New Task for Machine Comprehension of User Manuals](https://mlsanity.com/paper/2306.04187/) (2023)
- [SOPRAG: Multi-view Graph Experts Retrieval for Industrial Standard Operating Procedures](https://mlsanity.com/paper/2602.01858/) (2026)

## Abstract (arXiv)

Procedural text describes dynamic state changes during a step-by-step natural process (e.g., photosynthesis). In this work, we focus on the task of procedural text understanding, which aims to comprehend such documents and track entities' states and locations during a process. Although recent approaches have achieved substantial progress, their results are far behind human performance. Two challenges, the difficulty of commonsense reasoning and data insufficiency, still remain unsolved, which require the incorporation of external knowledge bases. Previous works on external knowledge injection usually rely on noisy web mining tools and heuristic rules with limited applicable scenarios. In this paper, we propose a novel KnOwledge-Aware proceduraL text understAnding (KOALA) model, which effectively leverages multiple forms of external knowledge in this task. Specifically, we retrieve informative knowledge triples from ConceptNet and perform knowledge-aware reasoning while tracking the entities. Besides, we employ a multi-stage training schema which fine-tunes the BERT model over unlabeled data collected from Wikipedia before further fine-tuning it on the final model. Experimental results on two procedural text datasets, ProPara and Recipes, verify the effectiveness of the proposed methods, in which our model achieves state-of-the-art performance in comparison to various baselines.