mlsanityData to 14 Oct 2026

Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition

arXiv 2410.01380 ↗First posted 2 Oct 2024cs.CL cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

In this work, we investigate how a model's tendency to broadly integrate its parametric knowledge evolves throughout pretraining, and how this behavior affects overall performance, particularly in terms of knowledge acquisition and forgetting. We introduce the concept of knowledge entropy, which quantifies the range of memory sources the model engages with; high knowledge entropy indicates that the model utilizes a wide range of memory sources, while low knowledge entropy suggests reliance on specific sources with greater certainty. Our analysis reveals a consistent decline in knowledge entropy as pretraining advances. We also find that the decline is closely associated with a reduction in the model's ability to acquire and retain knowledge, leading us to conclude that diminishing knowledge entropy (smaller number of active memory sources) impairs the model's knowledge acquisition and retention capabilities. We find further support for this by demonstrating that increasing the activity of inactive memory sources enhances the model's capacity for knowledge acquisition and retention.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

33 of 43 references.

References · 33

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2019Language Models as Knowledge Bases?cited by 1,936
  8. 2020GLU Variants Improve Transformercited by 1,804
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  11. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  12. 2024OLMo: Accelerating the Science of Language Modelscited by 613
21 more
  1. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  2. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  3. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  4. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  5. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  6. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  7. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  8. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  9. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  10. 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
  11. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 90
  12. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  13. 2019Calibration of Encoder Decoder Models for Neural Machine Translationcited by 68
  14. 2021Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?cited by 62
  15. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  16. 2023Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Modelscited by 18
  17. 2022Neural Knowledge Bank for Pretrained Transformerscited by 16
  18. 2022Entropy-based Stability-Plasticity for Lifelong Learningcited by 8
  19. 2023Online Continual Knowledge Learning for Language Modelscited by 6
  20. 2018Entropy in Quantum Information Theory -- Communication and Cryptographycited by 1
  21. 2024Entropy, Thermodynamics and the Geometrization of the Language Modelcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025The Price of Format: Diversity Collapse in LLMscited by 43
  2. 2025How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Trainingcited by 23
  3. 2025G2: Guided Generation for Enhanced Output Diversity in LLMscited by 14
  4. 2026Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMscited by 14
  5. 2025Learning Facts at Scale with Active Readingcited by 13
  6. 2025Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMscited by 11
  7. 2025Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimizationcited by 6
  8. 2025Train Once, Answer All: Many Pretraining Experiments for the Cost of Onecited by 6
  9. 2026Knowledge Vector Weakening: Efficient Training-free Unlearning for Large Vision-Language Modelscited by 3
  10. 2025Steering Information Utility in Key-Value Memory for Language Model Post-Trainingcited by 2
  11. 2026BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuningcited by 2
  12. 2026Continual Learning via Self-Probe Gradientscited by 1
4 more
  1. 2025Robust Federated Learning with Confidence-Weighted Filtering and GAN-Based Completion under Noisy and Incomplete Datacited by 0
  2. 2025Rewriting History: A Recipe for Interventional Analyses to Study Data Effects on Model Behaviorcited by 0
  3. 2026AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Trainingcited by 0
  4. 2026Specialists Learn Faster: Knowledge Organization in Pre-training Shapes Knowledge Update Efficiencycited by 0