mlsanityData to 7 Oct 2026

Dynamic Language Models for Continuously Evolving Content

arXiv 2106.06297 ↗First posted 11 Jun 2021cs.CL cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

The content on the web is in a constant state of flux. New entities, issues, and ideas continuously emerge, while the semantics of the existing conversation topics gradually shift. In recent years, pre-trained language models like BERT greatly improved the state-of-the-art for a large spectrum of content understanding tasks. Therefore, in this paper, we aim to study how these language models can be adapted to better handle continuously evolving web content. In our study, we first analyze the evolution of 2013 - 2019 Twitter data, and unequivocally confirm that a BERT model trained on past tweets would heavily deteriorate when directly applied to data from later years. Then, we investigate two possible sources of the deterioration: the semantic shift of existing tokens and the sub-optimal or failed understanding of new tokens. To this end, we both explore two different vocabulary composition methods, as well as propose three sampling methods which help in efficient incremental training for BERT-like models. Compared to a new model trained from scratch offline, our incremental training (a) reduces the training costs, (b) achieves better performance on evolving content, and (c) is suitable for online deployment. The superiority of our methods is validated using two downstream tasks. We demonstrate significant improvements when incrementally evolving the model from a particular base year, on the task of Country Hashtag Prediction, as well as on the OffensEval 2019 task.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

37 of 49 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  10. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  11. 2015Teaching Machines to Read and Comprehendcited by 1,039
  12. 2018End-to-End Incremental Learningcited by 596
25 more
  1. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  2. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
  3. 2017Incremental Learning of Object Detectors without Catastrophic Forgettingcited by 208
  4. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 206
  5. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  6. 2015Character-Aware Neural Language Modelscited by 129
  7. 2016Less-forgetting Learning in Deep Neural Networkscited by 97
  8. 2020The Cost of Training NLP Models: A Concise Overviewcited by 97
  9. 2018Diachronic word embeddings and semantic shifts: a surveycited by 90
  10. 2014Statistically Significant Detection of Linguistic Changecited by 82
  11. 2017Dynamic Word Embeddingscited by 51
  12. 2017Dynamic Word Embeddings for Evolving Semantic Discoverycited by 38
  13. 2016Gated-Attention Readers for Text Comprehensioncited by 37
  14. 2017Mimicking Word Embeddings using Subword RNNscited by 30
  15. 2019Few-Shot Representation Learning for Out-Of-Vocabulary Wordscited by 30
  16. 2015Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representationcited by 28
  17. 2019Improving Pre-Trained Multilingual Models with Vocabulary Expansioncited by 27
  18. 2018A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectorscited by 17
  19. 2016Charagram: Embedding Words and Sentences via Character n-gramscited by 14
  20. 2016Exponential Family Embeddingscited by 13
  21. 2017High-risk learning: acquiring new word vectors from tiny datacited by 13
  22. 2017Learning to Compute Word Embeddings On the Flycited by 12
  23. 2016Bayesian Neural Word Embeddingcited by 11
  24. 2016Vocabulary Manipulation for Neural Machine Translationcited by 5
  25. 2019Character n-gram Embeddings to Improve RNN Language Modelscited by 4

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 9; 2023: 4; 2024: 6; 2025: 3.

27 papers.

Sort:

By area:

  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  2. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  3. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  4. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 123
  5. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  6. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  7. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  8. 2021Time Masking for Temporal Language Modelscited by 46
  9. 2022Temporal Attention for Language Modelscited by 21
  10. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  11. 2022Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Changecited by 18
  12. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
15 more
  1. 2024ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Taskscited by 11
  2. 2022Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?cited by 9
  3. 2022The future is different: Large pre-trained language models fail in prediction taskscited by 8
  4. 2023Concept Drift Adaptation in Text Stream Mining Settings: A Systematic Reviewcited by 5
  5. 2024Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMscited by 5
  6. 2023A Massive Scale Semantic Similarity Dataset of Historical Englishcited by 4
  7. 2024Extracting Affect Aggregates from Longitudinal Social Media Data with Temporal Adapters for Large Language Modelscited by 4
  8. 2022Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptationcited by 2
  9. 2025TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammarcited by 2
  10. 2022Time-Aware Datasets are Adaptive Knowledgebases for the New Normalcited by 1
  11. 2023SMILE: Evaluation and Domain Adaptation for Social Media Language Understandingcited by 1
  12. 2025Iffy-Or-Not: Extending the Web to Support the Critical Evaluation of Fallacious Textscited by 1
  13. 2021Learning Neural Models for Natural Language Processing in the Face of Distributional Shiftcited by 0
  14. 2024Improving Sampling Methods for Fine-tuning SentenceBERT in Text Streamscited by 0
  15. 2024Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Modelcited by 0