mlsanityData to 14 Oct 2026

Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

arXiv 2309.14316 ↗First posted 25 Sep 2023cs.CL cs.AI cs.LGcited by 270

Not on the map. Why?

Cited by 43 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) can store a vast amount of world knowledge, often extractable via question-answering (e.g., "What is Abraham Lincoln's birthday?"). However, do they answer such questions based on exposure to similar questions during training (i.e., cheating), or by genuinely learning to extract knowledge from sources like Wikipedia? In this paper, we investigate this issue using a controlled biography dataset. We find a strong correlation between the model's ability to extract knowledge and various diversity measures of the training data. Essentially, for knowledge to be reliably extracted, it must be sufficiently augmented (e.g., through paraphrasing, sentence shuffling, translations) textitduring pretraining. Without such augmentation, knowledge may be memorized but not extractable, leading to 0% accuracy, regardless of subsequent instruction fine-tuning. To understand why this occurs, we employ (nearly) linear probing to demonstrate a strong connection between the observed correlation and how the model internally encodes knowledge -- whether it is linearly encoded in the hidden embeddings of entity names or distributed across other token embeddings in the training text. This paper provides several key recommendations for LLM pretraining in the industry: (1) rewrite the pretraining data -- using small, auxiliary models -- to provide knowledge augmentation, and (2) incorporate more instruction-finetuning data into the pretraining stage before it becomes too late.

Key facts. Cited by 270 papers in the map (data through 14 Oct 2026). Builds on 27 of its 43 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

27 of 43 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2019Language Models as Knowledge Bases?cited by 1,936
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  12. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
15 more
  1. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  2. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  3. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  4. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  5. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  6. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 168
  7. 2023Taken out of context: On measuring situational awareness in LLMscited by 143
  8. 2023Physics of Language Models: Part 1, Learning Hierarchical Language Structurescited by 83
  9. 2023ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbotscited by 47
  10. 2020Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweightcited by 34
  11. 2020The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understandingcited by 28
  12. 2022COPEN: Probing Conceptual Knowledge in Pre-trained Language Modelscited by 21
  13. 2019What Does My QA Model Know? Devising Controlled Probes using Expert Knowledgecited by 19
  14. 2021Inspecting the concept knowledge graph encoded by modern language modelscited by 15
  15. 2022ALBERT with Knowledge Graph Encoder Utilizing Semantic Similarity for Commonsense Question Answeringcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 77; 2025: 118; 2026: 68.

270 papers.

Sort:

By area:

  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  2. 2024Alignment faking in large language modelscited by 360
  3. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  4. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  5. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  6. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  7. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 130
  8. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 115
  9. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 108
  10. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
  11. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 101
  12. 2024Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Modelscited by 93
38 more
  1. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 90
  2. 2025Self-Adapting Language Modelscited by 84
  3. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 81
  4. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 78
  5. 2023Alignment for Honestycited by 74
  6. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 72
  7. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 69
  8. 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agentscited by 64
  9. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 63
  10. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 59
  11. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
  12. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 55
  13. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  14. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  15. 2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problemscited by 50
  16. 2024ICLR: In-Context Learning of Representationscited by 50
  17. 2024Reverse Training to Nurse the Reversal Cursecited by 49
  18. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 47
  19. 2024Theoretical limitations of multi-layer Transformercited by 45
  20. 2024Disentangling Memory and Reasoning Ability in Large Language Modelscited by 44
  21. 2024Changing Answer Order Can Decrease MMLU Accuracycited by 42
  22. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 41
  23. 2024Knowledge Boundary of Large Language Models: A Surveycited by 40
  24. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 40
  25. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 40
  26. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 39
  27. 2025Parametric Retrieval Augmented Generationcited by 38
  28. 2025How do language models learn facts? Dynamics, curricula and hallucinationscited by 37
  29. 2026Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformerscited by 37
  30. 2024InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoningcited by 36
  31. 2024Are large language models superhuman chemists?cited by 34
  32. 2024Scaling Laws for Fact Memorization of Large Language Modelscited by 33
  33. 2025Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Homecited by 33
  34. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 33
  35. 2024Teaching Large Language Models to Express Knowledge Boundary from Their Own Signalscited by 32
  36. 2024Understanding Finetuning for Factual Knowledge Extractioncited by 32
  37. 2024Counting Ability of Large Language Models and Impact of Tokenizationcited by 32
  38. 2025Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMscited by 31

50 most cited of 270 All 270 (JSON)