# Effective Use of Transformer Networks for Entity Tracking

arXiv: https://arxiv.org/abs/1909.02635 · HTML: https://mlsanity.com/paper/1909.02635/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2019.

## Built on by (most cited first, 11 of 11)

- [PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World](https://mlsanity.com/paper/2106.00188/) (2021)
- [Entity Tracking in Language Models](https://mlsanity.com/paper/2305.02363/) (2023)
- [Pre-train or Annotate? Domain Adaptation with a Constrained Budget](https://mlsanity.com/paper/2109.04711/) (2021)
- [Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding](https://mlsanity.com/paper/2109.04947/) (2021)
- [Coalescing Global and Local Information for Procedural Text Understanding](https://mlsanity.com/paper/2208.12848/) (2022)
- [LEMON: Language-Based Environment Manipulation via Execution-Guided Pre-training](https://mlsanity.com/paper/2201.08081/) (2022)
- [PizzaCommonSense: Learning to Model Commonsense Reasoning about Intermediate Steps in Cooking Recipes](https://mlsanity.com/paper/2401.06930/) (2024)
- [Procedural Text Understanding via Scene-Wise Evolution](https://mlsanity.com/paper/2203.07600/) (2022)
- [Entity Tracking via Effective Use of Multi-Task Learning Model and Mention-guided Decoding](https://mlsanity.com/paper/2210.06444/) (2022)
- [MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models](https://mlsanity.com/paper/2502.10886/) (2025)
- [Learning Complementary Action Modeling from Automotive Maintenance Instructions](https://mlsanity.com/paper/2606.27808/) (2026)

## Abstract (arXiv)

Tracking entities in procedural language requires understanding the transformations arising from actions on entities as well as those entities' interactions. While self-attention-based pre-trained language encoders like GPT and BERT have been successfully applied across a range of natural language understanding tasks, their ability to handle the nuances of procedural texts is still untested. In this paper, we explore the use of pre-trained transformer networks for entity tracking tasks in procedural text. First, we test standard lightweight approaches for prediction with pre-trained transformers, and find that these approaches underperform even simple baselines. We show that much stronger results can be attained by restructuring the input to guide the transformer model to focus on a particular entity. Second, we assess the degree to which transformer networks capture the process dynamics, investigating such factors as merged entities and oblique entity references. On two different tasks, ingredient detection in recipes and QA over scientific processes, we achieve state-of-the-art results, but our models still largely attend to shallow context clues and do not form complex representations of intermediate entity or process state.