mlsanityData to 14 Oct 2026

Time is Encoded in the Weights of Finetuned Language Models

arXiv 2312.13401 ↗First posted 20 Dec 2023cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

We present time vectors, a simple tool to customize language models to new time periods. Time vectors are created by finetuning a language model on data from a single time (e.g., a year or month), and then subtracting the weights of the original pretrained model. This vector specifies a direction in weight space that, as our experiments show, improves performance on text from that time period. Time vectors specialized to adjacent time periods appear to be positioned closer together in a manifold. Using this structure, we interpolate between time vectors to induce new models that perform better on intervening and future time periods, without any additional training. We demonstrate the consistency of our findings across different tasks, domains, model sizes, and time scales. Our results suggest that time is encoded in the weight space of finetuned models.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 20 Dec 2023.

Builds on

23 of 37 references.

References · 23

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  4. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  6. 2021Robust fine-tuning of zero-shot modelscited by 797
  7. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  8. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 294
  9. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  10. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 221
  11. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  12. 2022Patching open-vocabulary models by interpolating weightscited by 190
11 more
  1. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  2. 2022TimeLMs: Diachronic Language Models from Twittercited by 131
  3. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  4. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 90
  5. 2017Dynamic Word Embeddingscited by 51
  6. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  7. 2021Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corporacited by 21
  8. 2023Mitigating Temporal Misalignment by Discarding Outdated Factscited by 21
  9. 2022Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Changecited by 18
  10. 2022Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?cited by 9
  11. 2023Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarizationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 13; 2026: 9.

33 papers.

Sort:

By area:

  1. 2024Large Language Models Can Learn Temporal Reasoningcited by 111
  2. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 88
  3. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  4. 2025Time-R1: Towards Comprehensive Temporal Reasoning in LLMscited by 25
  5. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  6. 2024Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMscited by 15
  7. 2025What is a Number, That a Large Language Model May Know It?cited by 15
  8. 2025Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Informationcited by 15
  9. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 15
  10. 2025Subspace-Boosted Model Mergingcited by 9
  11. 2024ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domainscited by 8
  12. 2025Language Models Can Predict Their Own Behaviorcited by 8
21 more
  1. 2024Merge to Learn: Efficiently Adding Skills to Language Models with Model Mergingcited by 7
  2. 2025Around the World in 24 Hours: Probing LLM Knowledge of Time and Placecited by 7
  3. 2024From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Modelscited by 6
  4. 2025Line of Sight: On Linear Representations in VLLMscited by 6
  5. 2024Continuous Language Model Interpolation for Dynamic and Controllable Text Generationcited by 3
  6. 2024Gradient Localization Improves Lifelong Pretraining of Language Modelscited by 3
  7. 2025Analyzing Continuous Semantic Shifts with Diachronic Word Similarity Matricescited by 3
  8. 2025Discrete Minds in a Continuous World: Do Language Models Know Time Passes?cited by 2
  9. 2026Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Modelscited by 2
  10. 2025TARDIS: Mitigating Temporal Misalignment via Representation Steeringcited by 1
  11. 2026Do Language Models Consistently Encode the Current Year?cited by 1
  12. 2024Towards understanding evolution of science through language model seriescited by 0
  13. 2025Temporal Generalization: A Reality Checkcited by 0
  14. 2025Structured yet Bounded Temporal Understanding in Large Language Modelscited by 0
  15. 2026Parameter-Efficient Temporal Domain Generalizationcited by 0
  16. 2026The Dynamic Gist-Based Memory Model (DGMM): A Memory-Centric Architecture for Artificial Intelligencecited by 0
  17. 2026Temporal Preference Optimization for Unsupervised Retrievalcited by 0
  18. 2026PredAct-Bench: Benchmarking Tool-Augmented Dialogue under Controlled Tool Noisecited by 0
  19. 2026Doc2LoRA Provides Decodable Representations of Scientific Ideascited by 0
  20. 2026Engram Injection: A Study of Temporal Validity in Parametric Language-Model Memorycited by 0
  21. 2026LLM Evaluation is a Forecasting Problemcited by 0