# Hubble: a Model Suite to Advance the Study of LLM Memorization

arXiv: https://arxiv.org/abs/2510.19811 · HTML: https://mlsanity.com/paper/2510.19811/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- First posted on arXiv 22 Oct 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Built on by (most cited first, 17 of 17)

- [Extracting books from production language models](https://mlsanity.com/paper/2601.02671/) (2026)
- [Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention](https://mlsanity.com/paper/2605.29548/) (2026)
- [Gaperon: A Peppered English-French Generative Language Model Suite](https://mlsanity.com/paper/2510.25771/) (2025)
- [Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models](https://mlsanity.com/paper/2603.20957/) (2026)
- [(How) Learning Rates Regulate Catastrophic Overtraining](https://mlsanity.com/paper/2604.13627/) (2026)
- [Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE](https://mlsanity.com/paper/2603.11611/) (2026)
- [Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics](https://mlsanity.com/paper/2606.06533/) (2026)
- [Quantifying the Effect of Test Set Contamination on Generative Evaluations](https://mlsanity.com/paper/2601.04301/) (2026)
- [Memorization Dynamics in Knowledge Distillation for Language Models](https://mlsanity.com/paper/2601.15394/) (2026)
- [Scaling Participation in Modular AI Systems](https://mlsanity.com/paper/2606.07812/) (2026)
- [Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores](https://mlsanity.com/paper/2608.02985/) (2026)
- [Beyond Forgetting: Representation Misdirection Elicits Controllable Side Behaviors and Capabilities](https://mlsanity.com/paper/2601.21702/) (2026)
- [Protecting patient privacy in clinical foundation models: Technical and legal perspectives](https://mlsanity.com/paper/2608.07705/) (2026)
- [SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion](https://mlsanity.com/paper/2605.07482/) (2026)
- [Correcting test set contamination by spiking the training data](https://mlsanity.com/paper/2605.24818/) (2026)
- [Clinically Grounded Privacy Evaluation of Medical LMs](https://mlsanity.com/paper/2606.09590/) (2026)
- [How Learning Governs Unlearning across the Memorization-Generalization Spectrum](https://mlsanity.com/paper/2610.08577/) (2026)

## Abstract (arXiv)

We present Hubble, a suite of fully open-source large language models (LLMs) for the scientific study of LLM memorization. Hubble models come in standard and perturbed variants: standard models are pretrained on a large English corpus, and perturbed models are trained in the same way but with controlled insertion of text (e.g., book passages, biographies, and test sets) designed to emulate key memorization risks. Our core release includes 8 models -- standard and perturbed models with 1B or 8B parameters, pretrained on 100B or 500B tokens -- establishing that memorization risks are determined by the frequency of sensitive data relative to size of the training corpus (i.e., a password appearing once in a smaller corpus is memorized better than the same password in a larger corpus). Our release also includes 6 perturbed models with text inserted at different pretraining phases, showing that sensitive data without continued exposure can be forgotten. These findings suggest two best practices for addressing memorization risks: to dilute sensitive data by increasing the size of the training corpus, and to order sensitive data to appear earlier in training. Beyond these general empirical findings, Hubble enables a broad range of memorization research; for example, analyzing the biographies reveals how readily different types of private information are memorized. We also demonstrate that the randomized insertions in Hubble make it an ideal testbed for membership inference and machine unlearning, and invite the community to further explore, benchmark, and build upon our work.