mlsanityData to 7 Oct 2026

Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

arXiv 2404.05405 ↗First posted 8 Apr 2024cs.CL cs.AI cs.LGcited by 137

Not on the map. Why?

Abstract (arXiv)

Scaling laws describe the relationship between the size of language models and their capabilities. Unlike prior studies that evaluate a model's capability via loss or benchmarks, we estimate the number of knowledge bits a model stores. We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page. Through multiple controlled datasets, we establish that language models can and only can store 2 bits of knowledge per parameter, even when quantized to int8, and such knowledge can be flexibly extracted for downstream applications. Consequently, a 7B model can store 14B bits of knowledge, surpassing the English Wikipedia and textbooks combined based on our estimation. More broadly, we present 12 results on how (1) training duration, (2) model architecture, (3) quantization, (4) sparsity constraints such as MoE, and (5) data signal-to-noise ratio affect a model's knowledge storage capacity. Notable insights include: * The GPT-2 architecture, with rotary embedding, matches or even surpasses LLaMA/Mistral architectures in knowledge storage, particularly over shorter training durations. This arises because LLaMA/Mistral uses GatedMLP, which is less stable and harder to train. * Prepending training data with domain names (e.g., wikipedia.org) significantly increases a model's knowledge capacity. Language models can autonomously identify and prioritize domains rich in knowledge, optimizing their storage capacity.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

24 of 34 references.

References · 24

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2023Mistral 7Bcited by 3,065
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2020GLU Variants Improve Transformercited by 1,724
  10. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  11. 2023Textbooks Are All You Needcited by 561
  12. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
12 more
  1. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  2. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  3. 2023Scaling Data-Constrained Language Modelscited by 391
  4. 2021Scaling Laws for Transfercited by 265
  5. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  6. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  7. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  8. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 142
  9. 2023Physics of Language Models: Part 1, Learning Hierarchical Language Structurescited by 83
  10. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
  11. 2024Reverse Training to Nurse the Reversal Cursecited by 48
  12. 2021Scaling Laws for Deep Learningcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 61; 2026: 45.

137 papers.

Sort:

By area:

  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  2. 2024Alignment faking in large language modelscited by 316
  3. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  4. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 223
  5. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  6. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 107
  7. 2024Scaling Laws for Precisioncited by 92
  8. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  9. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 55
  10. 2025How much do language models memorize?cited by 53
  11. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  12. 2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problemscited by 49
38 more
  1. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  2. 2024Understanding Factual Recall in Transformers via Associative Memoriescited by 45
  3. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  4. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 38
  5. 2024Scaling Laws for Fact Memorization of Large Language Modelscited by 33
  6. 2025How do language models learn facts? Dynamics, curricula and hallucinationscited by 33
  7. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  8. 2024CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?cited by 29
  9. 2025How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmarkcited by 26
  10. 2025Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Modelscited by 25
  11. 2024Sloth: scaling laws for LLM skills to predict multi-benchmark performance across familiescited by 23
  12. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 23
  13. 2025How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Trainingcited by 22
  14. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalecited by 20
  15. 2025Metadata Conditioning Accelerates Language Model Pre-trainingcited by 20
  16. 2025FineMedLM-o1: Enhancing Medical Knowledge Reasoning Ability of LLM from Supervised Fine-Tuning to Test-Time Trainingcited by 19
  17. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 19
  18. 2024ALPINE: Unveiling the Planning Capability of Autoregressive Learning in Language Modelscited by 18
  19. 2024Algorithmic Capabilities of Random Transformerscited by 17
  20. 2024Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisitioncited by 15
  21. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  22. 2025Knowledge Updating? No More Model Editing! Just Selective Contextual Reasoningcited by 15
  23. 2025Data Mixing Can Induce Phase Transitions in Knowledge Acquisitioncited by 15
  24. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 14
  25. 2025Analyzing Memorization in Large Language Models through the Lens of Model Attributioncited by 14
  26. 2024What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insightscited by 13
  27. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  28. 2024Enabling LLM Knowledge Analysis via Extensive Materializationcited by 12
  29. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  30. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformercited by 12
  31. 2025Do we really have to filter out random noise in pre-training data for language models?cited by 11
  32. 2025Scalable Fingerprinting of Large Language Modelscited by 11
  33. 2025Pre-training Limited Memory Language Models with Internal and External Knowledgecited by 11
  34. 2026Dynamic Skill Lifecycle Management for Agentic Reinforcement Learningcited by 10
  35. 2024Investigating Symbolic Capabilities of Large Language Modelscited by 8
  36. 2025Spectral Scaling Laws in Language Models: How Effectively Do Feed-Forward Networks Use Their Latent Space?cited by 8
  37. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compressioncited by 8
  38. 2025Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levelscited by 7

50 most cited of 137 All 137 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression
  2. 2025On the Fundamental Limits of LLMs at Scale
  3. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
  4. 2024Can Language Models Act as Knowledge Bases at Scale?
  5. 2025Generalizing Scaling Laws for Dense and Sparse Large Language Models
  6. 2024Observational Scaling Laws and the Predictability of Language Model Performance
  7. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
  8. 2025Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recall