mlsanityData to 7 Oct 2026

Towards understanding evolution of science through language model series

arXiv 2409.09636 ↗First posted 15 Sep 2024cs.CL cs.CY cs.DLcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce AnnualBERT, a series of language models designed specifically to capture the temporal evolution of scientific text. Deviating from the prevailing paradigms of subword tokenizations and "one model to rule them all", AnnualBERT adopts whole words as tokens and is composed of a base RoBERTa model pretrained from scratch on the full-text of 1.7 million arXiv papers published until 2008 and a collection of progressively trained models on arXiv papers at an annual basis. We demonstrate the effectiveness of AnnualBERT models by showing that they not only have comparable performances in standard tasks but also achieve state-of-the-art performances on domain-specific NLP tasks as well as link prediction tasks in the arXiv citation network. We then utilize probing tasks to quantify the models' behavior in terms of representation learning and forgetting as time progresses. Our approach enables the pretrained models to not only improve performances on scientific text processing tasks but also to provide insights into the development of scientific discourse over time. The series of the models is available at https://huggingface.co/jd445/AnnualBERTs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 80 references that are in the map. First posted on arXiv 15 Sep 2024.

Builds on

43 of 80 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2017Inductive Representation Learning on Large Graphscited by 6,284
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  9. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2022Editing Models with Task Arithmeticcited by 1,076
  12. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
31 more
  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  3. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  4. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  5. 2023Language Models Represent Space and Timecited by 306
  6. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  7. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  8. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 203
  9. 2019Structural Scaffolds for Citation Intent Classification in Scientific Publicationscited by 129
  10. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  11. 2022ChemBERTa-2: Towards Chemical Foundation Modelscited by 123
  12. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  13. 2017SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publicationscited by 101
  14. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  15. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  16. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 82
  17. 2021Generalizing Graph Neural Networks on Out-Of-Distribution Graphscited by 69
  18. 2021MathBERT: A Pre-Trained Model for Mathematical Formula Understandingcited by 58
  19. 2019Stacking Models for Nearly Optimal Link Prediction in Complex Networkscited by 35
  20. 2023Time is Encoded in the Weights of Finetuned Language Modelscited by 31
  21. 2022The Diminishing Returns of Masked Language Models to Sciencecited by 22
  22. 2023Mitigating Temporal Misalignment by Discarding Outdated Factscited by 21
  23. 2021OAG-BERT: Towards A Unified Backbone Language Model For Academic Knowledge Servicescited by 20
  24. 2020BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognitioncited by 19
  25. 2021Self-Adaptive Training: Bridging Supervised and Self-Supervised Learningcited by 17
  26. 2020Neural Embeddings of Scholarly Periodicals Reveal Complex Disciplinary Organizationscited by 16
  27. 2022SsciBERT: A Pre-trained Language Model for Social Science Textscited by 12
  28. 2024A Survey of Pre-trained Language Models for Processing Scientific Textcited by 7
  29. 2019ScienceExamCER: A High-Density Fine-Grained Science-Domain Corpus for Common Entity Recognitioncited by 6
  30. 2022Pretraining without Wordpieces: Learning Over a Vocabulary of Millions of Wordscited by 6
  31. 2020Technological impact of biomedical research: the role of basicness and noveltycited by 2

Built on by

Not cited yet.