# Once Upon a textitTime in textitGraph: Relative-Time Pretraining for Complex Temporal Reasoning

arXiv: https://arxiv.org/abs/2310.14709 · HTML: https://mlsanity.com/paper/2310.14709/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 29 of 29)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [A Dataset for Answering Time-Sensitive Questions](https://mlsanity.com/paper/2108.06314/) (2021)
- [SituatedQA: Incorporating Extra-Linguistic Contexts into QA](https://mlsanity.com/paper/2109.06157/) (2021)
- [StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models](https://mlsanity.com/paper/2205.11388/) (2022)
- [Question Answering Over Temporal Knowledge Graphs](https://mlsanity.com/paper/2106.01515/) (2021)
- [Analysing Lexical Semantic Change with Contextualised Word Representations](https://mlsanity.com/paper/2004.14118/) (2020)
- [Time Waits for No One! Analysis and Challenges of Temporal Misalignment](https://mlsanity.com/paper/2111.07408/) (2021)
- [A Multi-Axis Annotation Scheme for Event Temporal Relations](https://mlsanity.com/paper/1804.07828/) (2018)
- [Complex Temporal Question Answering on Knowledge Graphs](https://mlsanity.com/paper/2109.08935/) (2021)
- [TIMEDIAL: Temporal Commonsense Reasoning in Dialog](https://mlsanity.com/paper/2106.04571/) (2021)
- [Dynamic Word Embeddings](https://mlsanity.com/paper/1702.08359/) (2017)
- [Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media](https://mlsanity.com/paper/2104.08116/) (2021)
- [TEQUILA: Temporal Question Answering over Knowledge Bases](https://mlsanity.com/paper/1908.03650/) (2019)
- [Time Masking for Temporal Language Models](https://mlsanity.com/paper/2110.06366/) (2021)
- [Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change](https://mlsanity.com/paper/1906.01688/) (2019)
- [ArchivalQA: A Large-scale Benchmark Dataset for Open Domain Question Answering over Historical News Collections](https://mlsanity.com/paper/2109.03438/) (2021)
- [Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change](https://mlsanity.com/paper/2210.17127/) (2022)
- [Extracting Temporal Event Relation with Syntax-guided Graph Transformer](https://mlsanity.com/paper/2104.09570/) (2021)
- [More than Classification: A Unified Framework for Event Temporal Relation Extraction](https://mlsanity.com/paper/2305.17607/) (2023)
- [Time-aware Prompting for Text Generation](https://mlsanity.com/paper/2211.02162/) (2022)

## Built on by (most cited first, 11 of 11)

- [Large Language Models Can Learn Temporal Reasoning](https://mlsanity.com/paper/2401.06853/) (2024)
- [Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning](https://mlsanity.com/paper/2406.09170/) (2024)
- [TimeArena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulation](https://mlsanity.com/paper/2402.05733/) (2024)
- [Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Models](https://mlsanity.com/paper/2402.16568/) (2024)
- [It's High Time: A Survey of Temporal Question Answering](https://mlsanity.com/paper/2505.20243/) (2025)
- [A Study into Investigating Temporal Robustness of LLMs](https://mlsanity.com/paper/2503.17073/) (2025)
- [Will LLMs Replace the Encoder-Only Models in Temporal Relation Classification?](https://mlsanity.com/paper/2410.10476/) (2024)
- [Transformer-Based Temporal Information Extraction and Application: A Review](https://mlsanity.com/paper/2504.07470/) (2025)
- [NeSTR: A Neuro-Symbolic Abductive Framework for Temporal Reasoning in Large Language Models](https://mlsanity.com/paper/2512.07218/) (2025)
- [LiFT: How to Enable In-Context Learning for Longitudinal Modelling](https://mlsanity.com/paper/2604.16382/) (2026)
- [Is the Future Predictable? Modeling Future Predictability for Reliable Temporal Reasoning](https://mlsanity.com/paper/2610.14073/) (2026)

## Abstract (arXiv)

Our physical world is constantly evolving over time, rendering challenges for pre-trained language models to understand and reason over the temporal contexts of texts. Existing work focuses on strengthening the direct association between a piece of text and its time-stamp. However, the knowledge-time association is usually insufficient for the downstream tasks that require reasoning over temporal dependencies between knowledge. In this work, we make use of the underlying nature of time, all temporally-scoped sentences are strung together through a one-dimensional time axis, and suggest creating a graph structure based on the relative placements of events along the time axis. Inspired by the graph view, we propose RemeMo (underlineRelative Tiunderlineme underlineModeling), which explicitly connects all temporally-scoped facts by modeling the time relations between any two sentences. Experimental results show that RemeMo outperforms the baseline T5 on multiple temporal question answering datasets under various settings. Further analysis suggests that RemeMo is especially good at modeling long-range complex temporal dependencies. We release our code and pre-trained checkpoints at hrefhttps://github.com/DAMO-NLP-SG/RemeMothis url.