# Time-Stamped Language Model: Teaching Language Models to Understand the Flow of Events

arXiv: https://arxiv.org/abs/2104.07635 · HTML: https://mlsanity.com/paper/2104.07635/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes](https://mlsanity.com/paper/1809.00812/) (2018)
- [Relational recurrent neural networks](https://mlsanity.com/paper/1806.01822/) (2018)
- [Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension](https://mlsanity.com/paper/1805.06975/) (2018)
- [Simulating Action Dynamics with Neural Process Networks](https://mlsanity.com/paper/1711.05313/) (2017)
- [Tracking the World State with Recurrent Entity Networks](https://mlsanity.com/paper/1612.03969/) (2016)
- [Simpler Context-Dependent Logical Forms via Model Projections](https://mlsanity.com/paper/1606.05378/) (2016)
- [Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text](https://mlsanity.com/paper/1909.04745/) (2019)
- [Reasoning about Actions and State Changes by Injecting Commonsense Knowledge](https://mlsanity.com/paper/1808.10012/) (2018)
- [Building Dynamic Knowledge Graphs from Text using Machine Reading Comprehension](https://mlsanity.com/paper/1810.05682/) (2018)
- [Knowledge-Aware Procedural Text Understanding with Multi-Stage Training](https://mlsanity.com/paper/2009.13199/) (2020)
- [Procedural Reading Comprehension with Attribute-Aware Context Flow](https://mlsanity.com/paper/2003.13878/) (2020)
- [Tracking Discrete and Continuous Entity State for Process Understanding](https://mlsanity.com/paper/1904.03518/) (2019)
- [Procedural Reasoning Networks for Understanding Multimodal Procedures](https://mlsanity.com/paper/1909.08859/) (2019)
- [Latent Alignment of Procedural Concepts in Multimodal Recipes](https://mlsanity.com/paper/2101.04727/) (2021)

## Built on by (most cited first, 22 of 22)

- [Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker](https://mlsanity.com/paper/2306.00924/) (2023)
- [Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models](https://mlsanity.com/paper/2306.08952/) (2023)
- [DomiKnowS: A Library for Integration of Symbolic Domain Knowledge in Deep Learning](https://mlsanity.com/paper/2108.12370/) (2021)
- [Coalescing Global and Local Information for Procedural Text Understanding](https://mlsanity.com/paper/2208.12848/) (2022)
- [GLUECons: A Generic Benchmark for Learning Under Constraints](https://mlsanity.com/paper/2302.10914/) (2023)
- [Understand the Dynamic World: An End-to-End Knowledge Informed Framework for Open Domain Entity State Tracking](https://mlsanity.com/paper/2304.13854/) (2023)
- [Enhancing Temporal Sensitivity and Reasoning for Time-Sensitive Question Answering](https://mlsanity.com/paper/2409.16909/) (2024)
- [Examining and Adapting Time for Multilingual Classification via Mixture of Temporal Experts](https://mlsanity.com/paper/2502.08825/) (2025)
- [CrisisLTLSum: A Benchmark for Local Crisis Event Timeline Extraction and Summarization](https://mlsanity.com/paper/2210.14190/) (2022)
- [LEMON: Language-Based Environment Manipulation via Execution-Guided Pre-training](https://mlsanity.com/paper/2201.08081/) (2022)
- [Dynamic Relevance Graph Network for Knowledge-Aware Question Answering](https://mlsanity.com/paper/2209.09947/) (2022)
- [The Role of Semantic Parsing in Understanding Procedural Text](https://mlsanity.com/paper/2302.06829/) (2023)
- [Prompt2DeModel: Declarative Neuro-Symbolic Modeling with Natural Language](https://mlsanity.com/paper/2407.20513/) (2024)
- [Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension](https://mlsanity.com/paper/2204.02566/) (2022)
- [Transferring Procedural Knowledge across Commonsense Tasks](https://mlsanity.com/paper/2304.13867/) (2023)
- [Consistent Joint Decision-Making with Heterogeneous Learning Models](https://mlsanity.com/paper/2402.03728/) (2024)
- [Order-Based Pre-training Strategies for Procedural Text Understanding](https://mlsanity.com/paper/2404.04676/) (2024)
- [Discrete Minds in a Continuous World: Do Language Models Know Time Passes?](https://mlsanity.com/paper/2506.05790/) (2025)
- [Procedural Text Understanding via Scene-Wise Evolution](https://mlsanity.com/paper/2203.07600/) (2022)
- [Entity Tracking via Effective Use of Multi-Task Learning Model and Mention-guided Decoding](https://mlsanity.com/paper/2210.06444/) (2022)
- [CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text](https://mlsanity.com/paper/2310.14326/) (2023)
- [Multilingual LLMs Inherently Reward In-Language Time-Sensitive Semantic Alignment for Low-Resource Languages](https://mlsanity.com/paper/2412.08090/) (2024)

## Abstract (arXiv)

Tracking entities throughout a procedure described in a text is challenging due to the dynamic nature of the world described in the process. Firstly, we propose to formulate this task as a question answering problem. This enables us to use pre-trained transformer-based language models on other QA benchmarks by adapting those to the procedural text understanding. Secondly, since the transformer-based language models cannot encode the flow of events by themselves, we propose a Time-Stamped Language Model~(TSLM model) to encode event information in LMs architecture by introducing the timestamp encoding. Our model evaluated on the Propara dataset shows improvements on the published state-of-the-art results with a 3.1% increase in F1 score. Moreover, our model yields better results on the location prediction task on the NPN-Cooking dataset. This result indicates that our approach is effective for procedural text understanding in general.