mlsanityData to 14 Oct 2026

ScienceMeter: Tracking Scientific Knowledge Updates in Language Models

arXiv 2505.24302 ↗First posted 30 May 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are increasingly used to support scientific research, but their knowledge of scientific advancements can quickly become outdated. We introduce ScienceMeter, a new framework for evaluating scientific knowledge update methods over scientific knowledge spanning the past, present, and future. ScienceMeter defines three metrics: knowledge preservation, the extent to which models' understanding of previously learned papers is preserved; knowledge acquisition, how well scientific claims from newly introduced papers are acquired; and knowledge projection, the ability of the updated model to anticipate or generalize to related scientific claims that may emerge in the future. Using ScienceMeter, we evaluate the scientific knowledge of LLMs through claim judgment and generation tasks on a curated dataset across ten domains. We evaluate five representative knowledge update approaches and find that the best-performing knowledge update methods can preserve only 85.9% of existing knowledge, acquire 71.7% of new knowledge, and project 37.7% of future knowledge, underscoring that developing robust scientific knowledge update mechanisms is both crucial and challenging.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 58 of its 65 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

58 of 65 references.

References · 58

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
38 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  4. 2022Mass-Editing Memory in a Transformercited by 1,001
  5. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  6. 2017Learning multiple visual domains with residual adapterscited by 557
  7. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  8. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  9. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  10. 20242 OLMo 2 Furiouscited by 340
  11. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  12. 2023Knowledge Editing for Large Language Models: A Surveycited by 194
  13. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  14. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
  15. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  16. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  17. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 119
  18. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  19. 2024Mapping the Increasing Use of LLMs in Scientific Paperscited by 118
  20. 2023The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4cited by 112
  21. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 100
  22. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  23. 2024WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Modelscited by 92
  24. 2025LLM4SR: A Survey on Large Language Models for Scientific Researchcited by 83
  25. 2024OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMscited by 72
  26. 2024LitLLM: A Toolkit for Scientific Literature Reviewcited by 63
  27. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 55
  28. 2023Resolving Knowledge Conflicts in Large Language Modelscited by 50
  29. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  30. 2023Large Language Models are Zero Shot Hypothesis Proposerscited by 45
  31. 2024Learning to Edit: Aligning LLMs with Knowledge Editingcited by 43
  32. 2024CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Supportcited by 29
  33. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 28
  34. 2024Set the Clock: Temporal Alignment of Pretrained Language Modelscited by 25
  35. 2023HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Sciencecited by 22
  36. 2024CodeUpdateArena: Benchmarking Knowledge Editing on API Updatescited by 19
  37. 2023History Matters: Temporal Knowledge Editing in Large Language Modelcited by 17
  38. 2024Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teachingcited by 17

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Don't Throw Away Your Pretrained Modelcited by 10
  2. 2026MoCo: A One-Stop Shop for Model Collaboration Researchcited by 6
  3. 2026Can Scientific Claims Be Removed from Large Language Models? A Systematic Evaluation of Claim-Level Unlearningcited by 0
  4. 2026PAPER2LLM++: Continual Self-Evolution of LLMs from Research Paperscited by 0