# TIMELINE: Exhaustive Annotation of Temporal Relations Supporting the Automatic Ordering of Events in News Articles

arXiv: https://arxiv.org/abs/2310.17802 · HTML: https://mlsanity.com/paper/2310.17802/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 30 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Multi-Axis Annotation Scheme for Event Temporal Relations](https://mlsanity.com/paper/1804.07828/) (2018)
- [An Improved Neural Baseline for Temporal Relation Extraction](https://mlsanity.com/paper/1909.00429/) (2019)
- [Deep Structured Neural Network for Event Temporal Relation Extraction](https://mlsanity.com/paper/1909.10094/) (2019)
- [CogCompTime: A Tool for Understanding Time in Natural Language Text](https://mlsanity.com/paper/1906.04940/) (2019)
- [A Survey on Event-based News Narrative Extraction](https://mlsanity.com/paper/2302.08351/) (2023)

## Built on by (most cited first, 8 of 8)

- [Will LLMs Replace the Encoder-Only Models in Temporal Relation Classification?](https://mlsanity.com/paper/2410.10476/) (2024)
- [Event Extraction in Large Language Model](https://mlsanity.com/paper/2512.19537/) (2025)
- [Beyond Pairwise: Global Zero-shot Temporal Graph Generation](https://mlsanity.com/paper/2502.11114/) (2025)
- [Formulation Comparison for Timeline Construction using LLMs](https://mlsanity.com/paper/2403.00990/) (2024)
- [EventFull: Complete and Consistent Event Relation Annotation](https://mlsanity.com/paper/2412.12733/) (2024)
- [The Temporal Game: A New Perspective on Temporal Relation Extraction](https://mlsanity.com/paper/2509.00250/) (2025)
- [Looking for the Bottleneck in Fine-grained Temporal Relation Classification](https://mlsanity.com/paper/2604.24620/) (2026)
- [Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives](https://mlsanity.com/paper/2610.17402/) (2026)

## Abstract (arXiv)

Temporal relation extraction models have thus far been hindered by a number of issues in existing temporal relation-annotated news datasets, including: (1) low inter-annotator agreement due to the lack of specificity of their annotation guidelines in terms of what counts as a temporal relation; (2) the exclusion of long-distance relations within a given document (those spanning across different paragraphs); and (3) the exclusion of events that are not centred on verbs. This paper aims to alleviate these issues by presenting a new annotation scheme that clearly defines the criteria based on which temporal relations should be annotated. Additionally, the scheme includes events even if they are not expressed as verbs (e.g., nominalised events). Furthermore, we propose a method for annotating all temporal relations -- including long-distance ones -- which automates the process, hence reducing time and manual effort on the part of annotators. The result is a new dataset, the TIMELINE corpus, in which improved inter-annotator agreement was obtained, in comparison with previously reported temporal relation datasets. We report the results of training and evaluating baseline temporal relation extraction models on the new corpus, and compare them with results obtained on the widely used MATRES corpus.