mlsanityData to 7 Oct 2026

Time-Stamped Language Model: Teaching Language Models to Understand the Flow of Events

arXiv 2104.07635 ↗First posted 15 Apr 2021cs.CL cs.AI cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Tracking entities throughout a procedure described in a text is challenging due to the dynamic nature of the world described in the process. Firstly, we propose to formulate this task as a question answering problem. This enables us to use pre-trained transformer-based language models on other QA benchmarks by adapting those to the procedural text understanding. Secondly, since the transformer-based language models cannot encode the flow of events by themselves, we propose a Time-Stamped Language Model~(TSLM model) to encode event information in LMs architecture by introducing the timestamp encoding. Our model evaluated on the Propara dataset shows improvements on the published state-of-the-art results with a 3.1% increase in F1 score. Moreover, our model yields better results on the location prediction task on the NPN-Cooking dataset. This result indicates that our approach is effective for procedural text understanding in general.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

22 of 30 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  7. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  8. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  9. 2018RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipescited by 74
  10. 2018Relational recurrent neural networkscited by 65
  11. 2018Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehensioncited by 62
  12. 2017Simulating Action Dynamics with Neural Process Networkscited by 53
10 more
  1. 2016Tracking the World State with Recurrent Entity Networkscited by 51
  2. 2016Simpler Context-Dependent Logical Forms via Model Projectionscited by 33
  3. 2019Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Textcited by 31
  4. 2018Reasoning about Actions and State Changes by Injecting Commonsense Knowledgecited by 29
  5. 2018Building Dynamic Knowledge Graphs from Text using Machine Reading Comprehensioncited by 17
  6. 2020Knowledge-Aware Procedural Text Understanding with Multi-Stage Trainingcited by 17
  7. 2020Procedural Reading Comprehension with Attribute-Aware Context Flowcited by 16
  8. 2019Tracking Discrete and Continuous Entity State for Process Understandingcited by 15
  9. 2019Procedural Reasoning Networks for Understanding Multimodal Procedurescited by 6
  10. 2021Latent Alignment of Procedural Concepts in Multimodal Recipescited by 2

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 7; 2023: 7; 2024: 5; 2025: 2.

22 papers.

Sort:

By area:

  1. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  2. 2023Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Modelscited by 37
  3. 2021DomiKnowS: A Library for Integration of Symbolic Domain Knowledge in Deep Learningcited by 12
  4. 2022Coalescing Global and Local Information for Procedural Text Understandingcited by 10
  5. 2023GLUECons: A Generic Benchmark for Learning Under Constraintscited by 9
  6. 2024Enhancing Temporal Sensitivity and Reasoning for Time-Sensitive Question Answeringcited by 9
  7. 2025Examining and Adapting Time for Multilingual Classification via Mixture of Temporal Expertscited by 9
  8. 2023Understand the Dynamic World: An End-to-End Knowledge Informed Framework for Open Domain Entity State Trackingcited by 8
  9. 2022CrisisLTLSum: A Benchmark for Local Crisis Event Timeline Extraction and Summarizationcited by 6
  10. 2022LEMON: Language-Based Environment Manipulation via Execution-Guided Pre-trainingcited by 5
  11. 2022Dynamic Relevance Graph Network for Knowledge-Aware Question Answeringcited by 4
  12. 2023The Role of Semantic Parsing in Understanding Procedural Textcited by 4
10 more
  1. 2024Prompt2DeModel: Declarative Neuro-Symbolic Modeling with Natural Languagecited by 3
  2. 2022Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehensioncited by 2
  3. 2023Transferring Procedural Knowledge across Commonsense Taskscited by 2
  4. 2024Order-Based Pre-training Strategies for Procedural Text Understandingcited by 2
  5. 2025Discrete Minds in a Continuous World: Do Language Models Know Time Passes?cited by 2
  6. 2022Procedural Text Understanding via Scene-Wise Evolutioncited by 1
  7. 2022Entity Tracking via Effective Use of Multi-Task Learning Model and Mention-guided Decodingcited by 1
  8. 2024Consistent Joint Decision-Making with Heterogeneous Learning Modelscited by 1
  9. 2023CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Textcited by 0
  10. 2024Multilingual LLMs Inherently Reward In-Language Time-Sensitive Semantic Alignment for Low-Resource Languagescited by 0