# tieval: An Evaluation Framework for Temporal Information Extraction Systems

arXiv: https://arxiv.org/abs/2301.04643 · HTML: https://mlsanity.com/paper/2301.04643/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 50 references that are in the map.
- First posted on arXiv 11 Jan 2023.

## Builds on (most cited first, 8 of 8)

- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- ["Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding](https://mlsanity.com/paper/1909.03065/) (2019)
- [A Multi-Axis Annotation Scheme for Event Temporal Relations](https://mlsanity.com/paper/1804.07828/) (2018)
- [TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions](https://mlsanity.com/paper/2005.00242/) (2020)
- [Joint Reasoning for Temporal and Causal Relations](https://mlsanity.com/paper/1906.04941/) (2019)
- [An Improved Neural Baseline for Temporal Relation Extraction](https://mlsanity.com/paper/1909.00429/) (2019)
- [Improving Temporal Relation Extraction with a Globally Acquired Statistical Resource](https://mlsanity.com/paper/1804.06020/) (2018)
- [A Survey on Temporal Reasoning for Temporal Information Extraction from Text (Extended Abstract)](https://mlsanity.com/paper/2005.06527/) (2020)

## Built on by (most cited first, 5 of 5)

- [An Overview Of Temporal Commonsense Reasoning and Acquisition](https://mlsanity.com/paper/2308.00002/) (2023)
- [Formulation Comparison for Timeline Construction using LLMs](https://mlsanity.com/paper/2403.00990/) (2024)
- [TEI2GO: A Multilingual Approach for Fast Temporal Expression Identification](https://mlsanity.com/paper/2403.16804/) (2024)
- [The Temporal Game: A New Perspective on Temporal Relation Extraction](https://mlsanity.com/paper/2509.00250/) (2025)
- [Looking for the Bottleneck in Fine-grained Temporal Relation Classification](https://mlsanity.com/paper/2604.24620/) (2026)

## Abstract (arXiv)

Temporal information extraction (TIE) has attracted a great deal of interest over the last two decades, leading to the development of a significant number of datasets. Despite its benefits, having access to a large volume of corpora makes it difficult when it comes to benchmark TIE systems. On the one hand, different datasets have different annotation schemes, thus hindering the comparison between competitors across different corpora. On the other hand, the fact that each corpus is commonly disseminated in a different format requires a considerable engineering effort for a researcher/practitioner to develop parsers for all of them. This constraint forces researchers to select a limited amount of datasets to evaluate their systems which consequently limits the comparability of the systems. Yet another obstacle that hinders the comparability of the TIE systems is the evaluation metric employed. While most research works adopt traditional metrics such as precision, recall, and F1, a few others prefer temporal awareness -- a metric tailored to be more comprehensive on the evaluation of temporal systems. Although the reason for the absence of temporal awareness in the evaluation of most systems is not clear, one of the factors that certainly weights this decision is the necessity to implement the temporal closure algorithm in order to compute temporal awareness, which is not straightforward to implement neither is currently easily available. All in all, these problems have limited the fair comparison between approaches and consequently, the development of temporal extraction systems. To mitigate these problems, we have developed tieval, a Python library that provides a concise interface for importing different corpora and facilitates system evaluation. In this paper, we present the first public release of tieval and highlight its most relevant features.