# ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning

arXiv: https://arxiv.org/abs/2012.15283 · HTML: https://mlsanity.com/paper/2012.15283/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2020.

## Built on by (most cited first, 30 of 38)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Continual Pre-Training of Large Language Models: How to (re)warm your model?](https://mlsanity.com/paper/2308.04014/) (2023)
- [Continual Pre-Training Mitigates Forgetting in Language and Vision](https://mlsanity.com/paper/2205.09357/) (2022)
- [Towards Lifelong Learning of Large Language Models: A Survey](https://mlsanity.com/paper/2406.06391/) (2024)
- [Test of Time: Instilling Video-Language Models with a Sense of Time](https://mlsanity.com/paper/2301.02074/) (2023)
- [Large Language Models-guided Dynamic Adaptation for Temporal Knowledge Graph Reasoning](https://mlsanity.com/paper/2405.14170/) (2024)
- [Improving Time Sensitivity for Question Answering over Temporal Knowledge Graphs](https://mlsanity.com/paper/2203.00255/) (2022)
- [LEMON: Lossless model expansion](https://mlsanity.com/paper/2310.07999/) (2023)
- [Extracting Temporal Event Relation with Syntax-guided Graph Transformer](https://mlsanity.com/paper/2104.09570/) (2021)
- [Go Back in Time: Generating Flashbacks in Stories with Event Temporal Prompts](https://mlsanity.com/paper/2205.01898/) (2022)
- [More than Classification: A Unified Framework for Event Temporal Relation Extraction](https://mlsanity.com/paper/2305.17607/) (2023)
- [It's High Time: A Survey of Temporal Question Answering](https://mlsanity.com/paper/2505.20243/) (2025)
- [A Study into Investigating Temporal Robustness of LLMs](https://mlsanity.com/paper/2503.17073/) (2025)
- [On the Evolution of Knowledge Graphs: A Survey and Perspective](https://mlsanity.com/paper/2310.04835/) (2023)
- [Continual Learning Under Language Shift](https://mlsanity.com/paper/2311.01200/) (2023)
- [Enhancing Temporal Sensitivity and Reasoning for Time-Sensitive Question Answering](https://mlsanity.com/paper/2409.16909/) (2024)
- [When Continue Learning Meets Multimodal Large Language Model: A Survey](https://mlsanity.com/paper/2503.01887/) (2025)
- [Controllable Text Generation for Open-Domain Creativity and Fairness](https://mlsanity.com/paper/2209.12099/) (2022)
- [Learning Action Conditions from Instructional Manuals for Instruction Understanding](https://mlsanity.com/paper/2205.12420/) (2022)
- [SimCLAD: A Simple Framework for Contrastive Learning of Acronym Disambiguation](https://mlsanity.com/paper/2111.14306/) (2021)
- [Are Fairy Tales Fair? Analyzing Gender Bias in Temporal Narrative Event Chains of Children's Fairy Tales](https://mlsanity.com/paper/2305.16641/) (2023)
- [Event Temporal Relation Extraction based on Retrieval-Augmented on LLMs](https://mlsanity.com/paper/2403.15273/) (2024)
- [Cross-sensor self-supervised training and alignment for remote sensing](https://mlsanity.com/paper/2405.09922/) (2024)
- [Continual Learning for Temporal-Sensitive Question Answering](https://mlsanity.com/paper/2407.12470/) (2024)
- [NECE: Narrative Event Chain Extraction Toolkit](https://mlsanity.com/paper/2208.08063/) (2022)
- [COPR: Continual Learning Human Preference through Optimal Policy Regularization](https://mlsanity.com/paper/2310.15694/) (2023)
- [Robust Uncertainty Quantification for Self-Evolving Large Language Models via Continual Domain Pretraining](https://mlsanity.com/paper/2510.22931/) (2025)
- [TemPrompt: Multi-Task Prompt Learning for Temporal Relation Extraction in RAG-based Crowdsourcing Systems](https://mlsanity.com/paper/2406.14825/) (2024)
- [NewsEdits 2.0: Learning the Intentions Behind Updating News](https://mlsanity.com/paper/2411.18811/) (2024)

## Abstract (arXiv)

While pre-trained language models (PTLMs) have achieved noticeable success on many NLP tasks, they still struggle for tasks that require event temporal reasoning, which is essential for event-centric applications. We present a continual pre-training approach that equips PTLMs with targeted knowledge about event temporal relations. We design self-supervised learning objectives to recover masked-out event and temporal indicators and to discriminate sentences from their corrupted counterparts (where event or temporal indicators got replaced). By further pre-training a PTLM with these objectives jointly, we reinforce its attention to event and temporal information, yielding enhanced capability on event temporal reasoning. This effective continual pre-training framework for event temporal reasoning (ECONET) improves the PTLMs' fine-tuning performances across five relation extraction and question answering tasks and achieves new or on-par state-of-the-art performances in most of our downstream tasks.