# BiTimeBERT: Extending Pre-Trained Language Representations with Bi-Temporal Information

arXiv: https://arxiv.org/abs/2204.13032 · HTML: https://mlsanity.com/paper/2204.13032/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 58 references that are in the map.
- First posted on arXiv 27 Apr 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis](https://mlsanity.com/paper/1904.02232/) (2019)
- [SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection](https://mlsanity.com/paper/2007.11464/) (2020)
- [Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model](https://mlsanity.com/paper/1912.09637/) (2019)
- [Analysing Lexical Semantic Change with Contextualised Word Representations](https://mlsanity.com/paper/2004.14118/) (2020)
- [Time Masking for Temporal Language Models](https://mlsanity.com/paper/2110.06366/) (2021)
- [Short-Term Meaning Shift: A Distributional Exploration](https://mlsanity.com/paper/1809.03169/) (2018)
- [Temporal Attention for Language Models](https://mlsanity.com/paper/2202.02093/) (2022)
- [A Temporally Sensitive Submodularity Framework for Timeline Summarization](https://mlsanity.com/paper/1810.07949/) (2018)

## Built on by (most cited first, 17 of 17)

- [TempRetriever: Fusion-based Temporal Dense Passage Retrieval for Time-Sensitive Questions](https://mlsanity.com/paper/2502.21024/) (2025)
- [It's High Time: A Survey of Temporal Question Answering](https://mlsanity.com/paper/2505.20243/) (2025)
- [A Study into Investigating Temporal Robustness of LLMs](https://mlsanity.com/paper/2503.17073/) (2025)
- [An Overview Of Temporal Commonsense Reasoning and Acquisition](https://mlsanity.com/paper/2308.00002/) (2023)
- [Temporal Blind Spots in Large Language Models](https://mlsanity.com/paper/2401.12078/) (2024)
- [ComplexTempQA:A 100m Dataset for Complex Temporal Question Answering](https://mlsanity.com/paper/2406.04866/) (2024)
- [Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models](https://mlsanity.com/paper/2406.01863/) (2024)
- [TempoFormer: A Transformer for Temporally-aware Representations in Change Detection](https://mlsanity.com/paper/2408.15689/) (2024)
- [Temporal Information Retrieval via Time-Specifier Model Merging](https://mlsanity.com/paper/2507.06782/) (2025)
- [Reading Between the Timelines: RAG for Answering Diachronic Questions](https://mlsanity.com/paper/2507.22917/) (2025)
- [Temporal Validity Change Prediction](https://mlsanity.com/paper/2401.00779/) (2024)
- [Discrete Minds in a Continuous World: Do Language Models Know Time Passes?](https://mlsanity.com/paper/2506.05790/) (2025)
- [Efficient Temporal-aware Matryoshka Adaptation for Temporal Information Retrieval](https://mlsanity.com/paper/2601.05549/) (2026)
- [Few-Shot Joint Multimodal Entity-Relation Extraction via Knowledge-Enhanced Cross-modal Prompt Model](https://mlsanity.com/paper/2410.14225/) (2024)
- [ChunQiuTR: Time-Keyed Temporal Retrieval in Classical Chinese Annals](https://mlsanity.com/paper/2604.06997/) (2026)
- [Temporal Preference Optimization for Unsupervised Retrieval](https://mlsanity.com/paper/2606.17664/) (2026)
- [TEMPS: Temporal Sentence Embeddings for Temporal Information Retrieval](https://mlsanity.com/paper/2609.28048/) (2026)

## Abstract (arXiv)

Time is an important aspect of documents and is used in a range of NLP and IR tasks. In this work, we investigate methods for incorporating temporal information during pre-training to further improve the performance on time-related tasks. Compared with common pre-trained language models like BERT which utilize synchronic document collections (e.g., BookCorpus and Wikipedia) as the training corpora, we use long-span temporal news article collection for building word representations. We introduce BiTimeBERT, a novel language representation model trained on a temporal collection of news articles via two new pre-training tasks, which harnesses two distinct temporal signals to construct time-aware language representations. The experimental results show that BiTimeBERT consistently outperforms BERT and other existing pre-trained models with substantial gains on different downstream NLP tasks and applications for which time is of importance (e.g., the accuracy improvement over BERT is 155% on the event time estimation task).