# Independence Tests for Language Models

arXiv: https://arxiv.org/abs/2502.12292 · HTML: https://mlsanity.com/paper/2502.12292/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 16 of 16)

- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [Llemma: An Open Language Model For Mathematics](https://mlsanity.com/paper/2310.10631/) (2023)
- [Robust Distortion-free Watermarks for Language Models](https://mlsanity.com/paper/2307.15593/) (2023)
- [Undetectable Watermarks for Language Models](https://mlsanity.com/paper/2306.09194/) (2023)
- [Compact Language Models via Pruning and Knowledge Distillation](https://mlsanity.com/paper/2407.14679/) (2024)
- [LLM360: Towards Fully Transparent Open-Source LLMs](https://mlsanity.com/paper/2312.06550/) (2023)
- [Instructional Fingerprinting of Large Language Models](https://mlsanity.com/paper/2401.12255/) (2024)
- [On the Societal Impact of Open Foundation Models](https://mlsanity.com/paper/2403.07918/) (2024)
- [REEF: Representation Encoding Fingerprints for Large Language Models](https://mlsanity.com/paper/2410.14273/) (2024)
- [A Fingerprint for Large Language Models](https://mlsanity.com/paper/2407.01235/) (2024)
- [PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks](https://mlsanity.com/paper/2404.04671/) (2024)
- [ProFLingo: A Fingerprinting-based Intellectual Property Protection Scheme for Large Language Models](https://mlsanity.com/paper/2405.02466/) (2024)
- [Model Provenance Testing for Large Language Models](https://mlsanity.com/paper/2502.00706/) (2025)

## Built on by (most cited first, 10 of 10)

- [LLM DNA: Tracing Model Evolution via Functional Representations](https://mlsanity.com/paper/2509.24496/) (2025)
- [Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs](https://mlsanity.com/paper/2506.14003/) (2025)
- [Blackbox Model Provenance via Palimpsestic Membership Inference](https://mlsanity.com/paper/2510.19796/) (2025)
- [Advertising in AI systems: Society must be vigilant](https://mlsanity.com/paper/2505.18425/) (2025)
- [Understanding Adversarial Transfer: Why Representation-Space Attacks Fail Where Data-Space Attacks Succeed](https://mlsanity.com/paper/2510.01494/) (2025)
- [One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMs](https://mlsanity.com/paper/2512.14751/) (2025)
- [Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs](https://mlsanity.com/paper/2606.12385/) (2026)
- [Correcting test set contamination by spiking the training data](https://mlsanity.com/paper/2605.24818/) (2026)
- [Witness Overlap: Directional Provenance Inside Open-Weight Model Families](https://mlsanity.com/paper/2609.31784/) (2026)
- [Training Witnesses: Trusting the Training without Trusting the Trainer](https://mlsanity.com/paper/2609.33915/) (2026)

## Abstract (arXiv)

We consider the following problem: given the weights of two models, can we test whether they were trained independently -- i.e., from independent random initializations? We consider two settings: constrained and unconstrained. In the constrained setting, we make assumptions about model architecture and training and propose a family of statistical tests that yield exact p-values with respect to the null hypothesis that the models are trained from independent random initializations. These p-values are valid regardless of the composition of either model's training data; we compute them by simulating exchangeable copies of each model under our assumptions and comparing various similarity measures of weights and activations between the original two models versus these copies. We report the p-values from these tests on pairs of 21 open-weight models (210 total pairs) and correctly identify all pairs of non-independent models. Our tests remain effective even if one model was fine-tuned for many tokens. In the unconstrained setting, where we make no assumptions about training procedures, can change model architecture, and allow for adversarial evasion attacks, the previous tests no longer work. Instead, we propose a new test which matches hidden activations between two models, and which is robust to adversarial transformations and to changes in model architecture. The test can also do localized testing: identifying specific non-independent components of models. Though we no longer obtain exact p-values from this, empirically we find it behaves as one and reliably identifies non-independent models. Notably, we can use the test to identify specific parts of one model that are derived from another (e.g., how Llama 3.1-8B was pruned to initialize Llama 3.2-3B, or shared layers between Mistral-7B and StripedHyena-7B), and it is even robust to retraining individual layers of either model from scratch.