mlsanityData to 7 Oct 2026

Time Machine GPT

arXiv 2404.18543 ↗First posted 29 Apr 2024cs.CL cs.CE cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are often trained on extensive, temporally indiscriminate text corpora, reflecting the lack of datasets with temporal metadata. This approach is not aligned with the evolving nature of language. Conventional methods for creating temporally adapted language models often depend on further pre-training static models on time-specific data. This paper presents a new approach: a series of point-in-time LLMs called Time Machine GPT (TiMaGPT), specifically designed to be nonprognosticative. This ensures they remain uninformed about future factual information and linguistic changes. This strategy is beneficial for understanding language evolution and is of critical importance when applying models in dynamic contexts, such as time-series forecasting, where foresight of future information can prove problematic. We provide access to both the models and training datasets.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 28 of its 54 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

28 of 54 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  12. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
16 more
  1. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
  2. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  3. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  4. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  5. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
  6. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  7. 2020FinBERT: A Pretrained Language Model for Financial Communicationscited by 161
  8. 2018Diachronic word embeddings and semantic shifts: a surveycited by 90
  9. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  10. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  11. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  12. 2019Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shiftcited by 37
  13. 2020Dynamic Contextualized Word Embeddingscited by 20
  14. 2022HistBERT: A Pre-trained Language Model for Diachronic Lexical Semantic Analysiscited by 11
  15. 2017Dynamic Bernoulli Embeddings for Language Evolutioncited by 6
  16. 2023Variation and Instability in Dialect-Based Embedding Spacescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 86
  2. 2025A Fast and Effective Solution to the Problem of Look-ahead Bias in LLMscited by 8
  3. 2025Pretraining Language Models for Diachronic Linguistic Change Discoverycited by 4
  4. 2025Fusing Narrative Semantics for Financial Volatility Forecastingcited by 3
  5. 2026MemGuard-Alpha: Limits of Membership Inference for Detecting and Filtering Memorization-Contaminated Signals in LLM-Based Financial Forecastingcited by 0
  6. 2026Forecasting With LLMs: Improved Generalization Through Feature Steeringcited by 0
  7. 2026Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scorescited by 0