mlsanityData to 7 Oct 2026

TimeLMs: Diachronic Language Models from Twitter

arXiv 2202.03829 ↗First posted 8 Feb 2022cs.CL cs.AIcited by 129

Not on the map. Why?

Abstract (arXiv)

Despite its importance, the time variable has been largely neglected in the NLP and language model literature. In this paper, we present TimeLMs, a set of language models specialized on diachronic Twitter data. We show that a continual learning strategy contributes to enhancing Twitter-based language models' capacity to deal with future and out-of-distribution tweets, while making them competitive with standardized and more monolithic benchmarks. We also perform a number of qualitative analyses showing how they cope with trends and peaks in activity involving specific named entities or concept drift.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Builds on 24 of its 40 references that are in the map. First posted on arXiv 8 Feb 2022.

Builds on

24 of 40 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  8. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  9. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  10. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  11. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  12. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
12 more
  1. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
  2. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 209
  3. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 206
  4. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  5. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  6. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  7. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  8. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  9. 2021Time Masking for Temporal Language Modelscited by 46
  10. 2021Temporal Effects on Pre-trained Models for Language Processing Taskscited by 46
  11. 2018Short-Term Meaning Shift: A Distributional Explorationcited by 23
  12. 2020Dynamic Contextualized Word Embeddingscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 22; 2023: 36; 2024: 38; 2025: 19; 2026: 14.

129 papers.

Sort:

By area:

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  2. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  3. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  4. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 153
  5. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  6. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  7. 2023Generative Language Models Exhibit Social Identity Biasescited by 107
  8. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  9. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 82
  10. 2022TweetNLP: Cutting-Edge Natural Language Processing for Social Mediacited by 71
  11. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  12. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
38 more
  1. 2023Bias Against 93 Stigmatized Groups in Masked Language Models and Downstream Sentiment Classification Taskscited by 51
  2. 2022TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twittercited by 44
  3. 2023Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systemscited by 37
  4. 2022CommunityLM: Probing Partisan Worldviews from Language Modelscited by 36
  5. 2023Time is Encoded in the Weights of Finetuned Language Modelscited by 31
  6. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  7. 2023Robust Hate Speech Detection in Social Media: A Cross-Dataset Empirical Evaluationcited by 28
  8. 2022Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shiftscited by 27
  9. 2024Set the Clock: Temporal Alignment of Pretrained Language Modelscited by 24
  10. 2023LEMON: Lossless model expansioncited by 23
  11. 2022Twitter Topic Classificationcited by 22
  12. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  13. 2022TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Mediacited by 19
  14. 2024NEO-BENCH: Evaluating Robustness of Large Language Models with Neologismscited by 19
  15. 2022On the Impact of Temporal Concept Drift on Model Explanationscited by 14
  16. 2023TempEL: Linking Dynamically Evolving and Newly Emerging Entitiescited by 14
  17. 2023A Large Language Model Approach to Educational Survey Feedback Analysiscited by 14
  18. 2022Understanding Aesthetics with Language: A Photo Critique Dataset for Aesthetic Assessmentcited by 13
  19. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 13
  20. 2024ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Taskscited by 11
  21. 2023Can Word Sense Distribution Detect Semantic Changes of Words?cited by 10
  22. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignmentcited by 10
  23. 2022Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?cited by 9
  24. 2023Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarizationcited by 9
  25. 2023SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Researchcited by 9
  26. 2023Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognitioncited by 9
  27. 2023Continual Learning Under Language Shiftcited by 9
  28. 2022Using Twitter Data to Understand Public Perceptions of Approved versus Off-label Use for COVID-19-related Medicationscited by 8
  29. 2023Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddingscited by 8
  30. 2024CBF-LLM: Safe Control for LLM Alignmentcited by 8
  31. 2025CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experimentscited by 8
  32. 2024Reading Between the Tweets: Deciphering Ideological Stances of Interconnected Mixed-Ideology Communitiescited by 7
  33. 2024Prompt-Based Bias Calibration for Better Zero/Few-Shot Learning of Language Modelscited by 7
  34. 2024Automatically Labeling Clinical Trial Outcomes: A Large-Scale Benchmark for Drug Developmentcited by 7
  35. 2024Self-interpreting Adversarial Imagescited by 7
  36. 2024Divide-Verify-Refine: Can LLMs Self-Align with Complex Instructions?cited by 7
  37. 2023Public Attitudes Toward ChatGPT on Twitter: Sentiments, Topics, and Occupationscited by 6
  38. 2023textitSwap and Predict -- Predicting the Semantic Changes in Words across Corpora by Context Swappingcited by 6

50 most cited of 129 All 129 (JSON)