# BERT got a Date: Introducing Transformers to Temporal Tagging

arXiv: https://arxiv.org/abs/2109.14927 · HTML: https://mlsanity.com/paper/2109.14927/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 41 references that are in the map.
- First posted on arXiv 30 Sep 2021.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [Portuguese Named Entity Recognition using BERT-CRF](https://mlsanity.com/paper/1909.10649/) (2019)
- [CogCompTime: A Tool for Understanding Time in Natural Language Text](https://mlsanity.com/paper/1906.04940/) (2019)
- [Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text](https://mlsanity.com/paper/2005.09392/) (2020)
- [BERT-Based Multi-Head Selection for Joint Entity-Relation Extraction](https://mlsanity.com/paper/1908.05908/) (2019)

## Built on by (most cited first, 4 of 4)

- [ScatterShot: Interactive In-context Example Curation for Text Transformation](https://mlsanity.com/paper/2302.07346/) (2023)
- [Transformer-Based Temporal Information Extraction and Application: A Review](https://mlsanity.com/paper/2504.07470/) (2025)
- [CABACE: Injecting Character Sequence Information and Domain Knowledge for Enhanced Acronym and Long-Form Extraction](https://mlsanity.com/paper/2112.13237/) (2021)
- [A Modular Approach for Multilingual Timex Detection and Normalization using Deep Learning and Grammar-based methods](https://mlsanity.com/paper/2304.14221/) (2023)

## Abstract (arXiv)

Temporal expressions in text play a significant role in language understanding and correctly identifying them is fundamental to various retrieval and natural language processing systems. Previous works have slowly shifted from rule-based to neural architectures, capable of tagging expressions with higher accuracy. However, neural models can not yet distinguish between different expression types at the same level as their rule-based counterparts. In this work, we aim to identify the most suitable transformer architecture for joint temporal tagging and type classification, as well as, investigating the effect of semi-supervised training on the performance of these systems. Based on our study of token classification variants and encoder-decoder architectures, we present a transformer encoder-decoder model using the RoBERTa language model as our best performing system. By supplementing training resources with weakly labeled data from rule-based systems, our model surpasses previous works in temporal tagging and type classification, especially on rare classes. Our code and pre-trained experiments are available at: https://github.com/satya77/Transformer_Temporal_Tagger