Capability Emergence Can Be Forecast: Per-Seed, In Advance, With Calibrated Intervals, Certified False Alarms, and a Blind Pre-Registered Gate
Not on the map. Why?
Abstract (arXiv)
Emergent capabilities are widely treated as unpredictable: loss improves smoothly while abilities appear abruptly. Prior work offers early-warning indicators but never scores them as forecasts: no lead time at controlled false-alarm rate, no calibration, no negatives, no blind tests. We supply that discipline and show that, in grokking model systems and small language models, emergence timing is forecastable per run, in advance, with calibrated uncertainty. Across 30 transformers at identical configuration, the formation time of the previous-token head forecasts each seed's induction-head emergence at Spearman rho=0.977 with median lead 975 steps (~15% of training); a best-case loss rule ties the ranking with 50-step lead (a nowcast). Conformal intervals covered 15/15 held-out seeds, and the frozen rule passed blind pre-registered gates on TWO never-seen configurations (10/10 and 9/10 coverage). A trap-language rung then attacked our own rule as pre-registered: where previous-token context pays for the task itself, the bare precursor false-alarms on 10/10 capability-blocked runs, while the mechanism-composed conjunction is certified in both language classes (0 false alarms) and times emergence at rho=1.000. Finally, a gap-origin study broke the fixed offset (both lr and batch move the gap ~2.3x; no external clock owns it) and revealed the law beneath: across 80 valid-anchor runs the anchor fires at 0.843 of time-to-emergence -- t_event ~= 1.19 x t_anchor -- and this multiplicative rule passed its own blind gate (5/5) at a third unseen configuration. False alarms are certified against 33 manufactured negatives. The precursor leads across 3 public model families (Pythia, OLMo, OLMo-2; 7 suites), with OLMo-2 at 1B tokens showing precursor formed, capability absent. Four pre-registered kill criteria fired and are reported. Every freeze precedes its data in a public commit chain.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 17 references that are in the map. First posted on arXiv 17 Jul 2026.
Builds on
15 of 17 references.
References · 15
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
- 2022In-context Learning and Induction Headscited by 964
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
- 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
- 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
- 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
- 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 40
- 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 35
- 2023Predicting Emergent Abilities with Infinite Resolution Evaluationcited by 28
- 2024Predicting Emergent Capabilities by Finetuningcited by 10
- 2026From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretabilitycited by 4
3 more
Built on by
Not cited yet.