# Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Data

arXiv: https://arxiv.org/abs/2506.14704 · HTML: https://mlsanity.com/paper/2506.14704/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 17 Jun 2025.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023)
- [What Matters in Transformers? Not All Attention is Needed](https://mlsanity.com/paper/2406.15786/) (2024)
- [A Study on ReLU and Softmax in Transformer](https://mlsanity.com/paper/2302.06461/) (2023)
- [Memorization Capacity of Multi-Head Attention in Transformers](https://mlsanity.com/paper/2306.02010/) (2023)
- [Strategic Data Ordering: Enhancing Large Language Model Performance through Curriculum Learning](https://mlsanity.com/paper/2405.07490/) (2024)
- [Transcoders Beat Sparse Autoencoders for Interpretability](https://mlsanity.com/paper/2501.18823/) (2025)
- [Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining](https://mlsanity.com/paper/2502.06733/) (2025)
- [Breaking through the learning plateaus of in-context learning in Transformer](https://mlsanity.com/paper/2309.06054/) (2023)
- [SurgBox: Agent-Driven Operating Room Sandbox with Surgery Copilot](https://mlsanity.com/paper/2412.05187/) (2024)
- [Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks](https://mlsanity.com/paper/2305.15912/) (2023)
- [ChunkFormer: Learning Long Time Series with Multi-stage Chunked Transformer](https://mlsanity.com/paper/2112.15087/) (2021)
- [Comprehensive Modeling and Question Answering of Cancer Clinical Practice Guidelines using LLMs](https://mlsanity.com/paper/2501.13984/) (2025)
- [The Disharmony between BN and ReLU Causes Gradient Explosion, but is Offset by the Correlation between Activations](https://mlsanity.com/paper/2304.11692/) (2023)
- [Empirical Capacity Model for Self-Attention Neural Networks](https://mlsanity.com/paper/2407.15425/) (2024)

## Abstract (arXiv)

This paper studies how the model architecture and data configurations influence the empirical memorization capacity of generative transformers. The models are trained using synthetic text datasets derived from the Systematized Nomenclature of Medicine (SNOMED) knowledge graph: triplets, representing static connections, and sequences, simulating complex relation patterns. The results show that embedding size is the primary determinant of learning speed and capacity, while additional layers provide limited benefits and may hinder performance on simpler datasets. Activation functions play a crucial role, and Softmax demonstrates greater stability and capacity. Furthermore, increasing the complexity of the data set seems to improve the final memorization. These insights improve our understanding of transformer memory mechanisms and provide a framework for optimizing model design with structured real-world data.