mlsanityData to 7 Oct 2026

Time Masking for Temporal Language Models

arXiv 2110.06366 ↗First posted 12 Oct 2021cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

Our world is constantly evolving, and so is the content on the web. Consequently, our languages, often said to mirror the world, are dynamic in nature. However, most current contextual language models are static and cannot adapt to changes over time. In this work, we propose a temporal contextual language model called TempoBERT, which uses time as an additional context of texts. Our technique is based on modifying texts with temporal information and performing time masking - specific masking for the supplementary time information. We leverage our approach for the tasks of semantic change detection and sentence time prediction, experimenting on diverse datasets in terms of time, size, genre, and language. Our extensive evaluation shows that both tasks benefit from exploiting time masking.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

27 of 46 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2019Language Models as Knowledge Bases?cited by 1,922
  5. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  6. 2019How Can We Know What Language Models Know?cited by 953
  7. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  8. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
  9. 2020SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detectioncited by 105
  10. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  11. 2020Analysing Lexical Semantic Change with Contextualised Word Representationscited by 92
  12. 2018Diachronic word embeddings and semantic shifts: a surveycited by 90
15 more
  1. 2014Statistically Significant Detection of Linguistic Changecited by 82
  2. 2018Survey of Computational Approaches to Lexical Semantic Changecited by 60
  3. 2014Temporal Analysis of Language through Neural Language Modelscited by 53
  4. 2017Dynamic Word Embeddingscited by 51
  5. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  6. 2017Dynamic Word Embeddings for Evolving Semantic Discoverycited by 38
  7. 2019Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shiftcited by 37
  8. 2021Explaining and Improving BERT Performance on Lexical Semantic Change Detectioncited by 33
  9. 2019Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Changecited by 32
  10. 2019A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domainscited by 31
  11. 2021Dynamic Language Models for Continuously Evolving Contentcited by 27
  12. 2018Short-Term Meaning Shift: A Distributional Explorationcited by 23
  13. 2020Capturing Evolution in Word Usage: Just Add More Clusters?cited by 22
  14. 2019Dating Documents using Graph Convolution Networkscited by 9
  15. 2019AD3: Attentive Deep Document Datercited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 11; 2024: 15; 2025: 8; 2026: 3.

46 papers.

Sort:

By area:

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  2. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  3. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  4. 2024Large Language Models Can Learn Temporal Reasoningcited by 110
  5. 2022Temporal Attention for Language Modelscited by 21
  6. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  7. 2023A Survey on Contextualised Semantic Shift Detectioncited by 21
  8. 2024A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Changecited by 21
  9. 2022TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Mediacited by 19
  10. 2022BiTimeBERT: Extending Pre-Trained Language Representations with Bi-Temporal Informationcited by 17
  11. 2022Geographic Adaptation of Pretrained Language Modelscited by 14
  12. 2025TempRetriever: Fusion-based Temporal Dense Passage Retrieval for Time-Sensitive Questionscited by 14
34 more
  1. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 13
  2. 2025It's High Time: A Survey of Temporal Question Answeringcited by 12
  3. 2024ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Taskscited by 11
  4. 2023An Overview Of Temporal Commonsense Reasoning and Acquisitioncited by 10
  5. 2023Can Word Sense Distribution Detect Semantic Changes of Words?cited by 10
  6. 2023Once Upon a textitTime in textitGraph: Relative-Time Pretraining for Complex Temporal Reasoningcited by 10
  7. 2025A Study into Investigating Temporal Robustness of LLMscited by 10
  8. 2023Incorporating Structured Sentences with Time-enhanced BERT for Fully-inductive Temporal Relation Predictioncited by 9
  9. 2023Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarizationcited by 9
  10. 2023Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddingscited by 8
  11. 2024Temporal Blind Spots in Large Language Modelscited by 7
  12. 2024Analyzing Semantic Change through Lexical Replacementscited by 7
  13. 2023textitSwap and Predict -- Predicting the Semantic Changes in Words across Corpora by Context Swappingcited by 6
  14. 2024A Semantic Distance Metric Learning approach for Lexical Semantic Change Detectioncited by 5
  15. 2024Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Modelscited by 5
  16. 2024TempoFormer: A Transformer for Temporally-aware Representations in Change Detectioncited by 5
  17. 2025Temporal Information Retrieval via Time-Specifier Model Mergingcited by 5
  18. 2023Blind Dates: Examining the Expression of Temporality in Historical Photographscited by 4
  19. 2024Investigating the Contextualised Word Embedding Dimensions Specified for Contextual and Temporal Semantic Changescited by 4
  20. 2025SCDTour: Embedding Axis Ordering and Merging for Interpretable Semantic Change Detectioncited by 4
  21. 2025Analyzing Continuous Semantic Shifts with Diachronic Word Similarity Matricescited by 3
  22. 2022Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptationcited by 2
  23. 2022Is It Worth the (Environmental) Cost? Limited Evidence for Temporal Adaptation via Continuous Trainingcited by 2
  24. 2023Computational modeling of semantic changecited by 2
  25. 2024Temporal Validity Change Predictioncited by 2
  26. 2025TiMoE: Time-Aware Mixture of Language Expertscited by 2
  27. 2024Tracking Semantic Change in Slovene: A Novel Dataset and Optimal Transport-Based Distancecited by 1
  28. 2022Metadata Might Make Language Models Bettercited by 0
  29. 2024Future Language Modeling from Temporal Document Historycited by 0
  30. 2024A Systematic Analysis on the Temporal Generalization of Language Models in Social Mediacited by 0
  31. 2024Revealing COVID-19's Social Dynamics: Diachronic Semantic Analysis of Vaccine and Symptom Discourse on Twittercited by 0
  32. 2026Temporal Preference Optimization for Unsupervised Retrievalcited by 0
  33. 2026A Study of Temporal Fusion Strategies for Named Entity Recognition in Historical Textscited by 0
  34. 2026Recovering Temporal and Geographic Signals from Language Model Embeddingscited by 0