# What Makes a Representational Prior Work? Feature Families, Label-Free Invariances, and Critical Windows in Grokking

arXiv: https://arxiv.org/abs/2607.12735 · HTML: https://mlsanity.com/paper/2607.12735/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 4 references that are in the map.
- First posted on arXiv 14 Jul 2026.

## Builds on (most cited first, 4 of 4)

- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [The Weight Norm Sets the Grokking Timescale: A Causal Delay Law](https://mlsanity.com/paper/2606.13753/) (2026)

## Built on by (most cited first, 2 of 2)

- [Capability Emergence Can Be Forecast: Per-Seed, In Advance, With Calibrated Intervals, Certified False Alarms, and a Blind Pre-Registered Gate](https://mlsanity.com/paper/2609.19000/) (2026)
- [Small Enough to Know Everything: The Fully-Enumerable Transformer as an Instrument for the Science of Delayed Generalization](https://mlsanity.com/paper/2609.20166/) (2026)

## Abstract (arXiv)

Companion work showed the grokking delay is causally the time to form task-structured representations, injectable via a contrastive prior. Here we characterize what makes such a prior work, across four axes, in 188 new runs. Content: a coherent, learnable prior built from the wrong feature family (magnitude bands) blocks generalization like a random partition (1/15 vs 0/20 grok; p=0.43 between them), confirming the companion's prediction that priors act at the level of the circuit's features. Supervision: a fully label-free invariance prior -- positives are commuted pairs (a,b)~(b,a) only -- generalizes in 15/15 runs at a median 2.7× speedup, more reliably than the label-supervised prior itself (p=0.038), and combined with a weight-norm clamp yields the strongest method we test (median 17×, 5/5) -- strongest meaning reliably fast: plain cross-entropy with a clamp matches this speed only at the exact critical norm, while the prior keeps it fast across the entire clamp range. Timing: the prior is only needed early -- applied solely during the first 2000 epochs (4% of budget) it generalizes 10/10 at 2.7×, beating continuous application (8/10, 1.25×) and a duration-matched later window (2.1×). Setting: the dissociation replicates on modular multiplication and across depths and normalization variants, and a clamp sweep quantifies the companion's central claim: structure injection flattens the weight-norm delay-law exponent about 17-fold (plain cross-entropy slows 31× per +10 norm units, a lower bound as higher cells are censored, versus 1.22× with the prior). Honest boundary: tasks that generalize before memorizing have no delay to control. Feature-family alignment decides whether a prior permits generalization; invariance content suffices for acceleration without labels; a brief early window captures nearly all of the benefit.