# Pretraining with hierarchical memories: separating long-tail and common knowledge arXiv: https://arxiv.org/abs/2510.02375 ยท HTML: https://mlsanity.com/paper/2510.02375/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Builds on 54 of its 62 references that are in the map. - First posted on arXiv 29 Sep 2025. ## Builds on (most cited first, 30 of 54) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024) - [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018) - [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021) - [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016) - [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019) - [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021) - [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021) - [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017) - [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023) - [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022) - [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017) - [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019) - [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022) - [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019) - [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018) - [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023) - [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025) - [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020) - [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020) - [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021) - [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016) - [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022) - [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020) - [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022) - [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023) ## Built on by (most cited first, 14 of 14) - [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025) - [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026) - [Procedural Pretraining: Warming Up Language Models with Abstract Data](https://mlsanity.com/paper/2601.21725/) (2026) - [Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts](https://mlsanity.com/paper/2604.08519/) (2026) - [On Memory: A comparison of memory mechanisms in world models](https://mlsanity.com/paper/2512.06983/) (2025) - [MemAdapter: Fast Alignment across Agent Memory Paradigms via Generative Subgraph Retrieval](https://mlsanity.com/paper/2602.08369/) (2026) - [Optimal Splitting of Language Models from Mixtures to Specialized Domains](https://mlsanity.com/paper/2603.19149/) (2026) - [Protecting patient privacy in clinical foundation models: Technical and legal perspectives](https://mlsanity.com/paper/2608.07705/) (2026) - [EvoIR-Agent: Self-Evolving Image Restoration Agentic System via Experience-Driven Learning](https://mlsanity.com/paper/2605.22208/) (2026) - [Knowledge Offloading: Decomposing LLMs into Sparse Backbones and Memory Modules](https://mlsanity.com/paper/2605.29075/) (2026) - [Co-LMLM: Continuous-Query Limited Memory Language Models](https://mlsanity.com/paper/2607.07707/) (2026) - [BCMT: Blockwise Causal Memory Transformer](https://mlsanity.com/paper/2608.13578/) (2026) - [Co-Evolving Structured Knowledge and Reasoning in Language Models](https://mlsanity.com/paper/2608.26386/) (2026) - [Evaluating LLM Creativity through Long-Tail Performance](https://mlsanity.com/paper/2610.12998/) (2026) ## Abstract (arXiv) The impressive performance gains of modern language models currently rely on scaling parameters: larger models store more world knowledge and reason better. Yet compressing all world knowledge into parameters is unnecessary, as only a fraction is used per prompt, and impractical for edge devices with limited inference-time memory and compute. We address this shortcoming by a memory-augmented architecture and a pretraining strategy aligned with existing hardware paradigms. We introduce small language models that access large hierarchical parametric memory banks encoding world knowledge. During pretraining and inference, we fetch a small, context-dependent memory block and add it to the model. Our pretraining learns to store long-tail world knowledge in the memory parameters, while the small language model acts as an anchor capturing common knowledge and general reasoning abilities. Through trillion-token-scale experiments, we show significant gains: a 160M-parameters model augmented with an 18M-parameters memory fetched from a 4.6B memory bank obtains comparable performance to a regular model with more than 2x the parameters. Through extensive experiments, we study the optimal type and size of parametric memories in transformers, scaling them to over 21B parameters. We find that our proposed hierarchical feed-forward memories work robustly across transformer architectures, whether added during pretraining or post-hoc.