Marginals Before Conditionals
Not on the map. Why?
Abstract (arXiv)
We construct a minimal task that isolates conditional learning in neural networks: a surjective map with K-fold ambiguity, resolved by a selector token z, so H(A | B) = log K while H(A | B, z) = 0. The model learns the marginal P(A | B) first, producing a plateau at exactly log K, before acquiring the full conditional in a sharp, collective transition. The plateau has a clean decomposition: height = log K (set by ambiguity), duration = f(D) (set by dataset size D, not K). Gradient noise stabilizes the marginal solution: higher learning rates monotonically slow the transition (3.6* across a 7* η range at fixed throughput), and batch-size reduction delays escape, consistent with an entropic force opposing departure from the low-gradient marginal. Internally, a selector-routing head assembles during the plateau, leading the loss transition by ~50% of the waiting time. This is the Type 2 directional asymmetry of Papadopoulos et al. [2024], measured dynamically: we track the excess risk from log K to zero and characterize what stabilizes it, what triggers its collapse, and how long it takes.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 13 references that are in the map. First posted on arXiv 10 Mar 2026.
Builds on
12 of 13 references.
References · 12
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
- 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
- 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
- 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
- 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 43
- 2023Symmetry Induces Structure and Constraint of Learningcited by 25
- 2024The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and Morecited by 25
- 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
- 2024Arrows of Time for Large Language Modelscited by 18
- 2023A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descentcited by 8
- 2025Adaptive Heavy-Tailed Stochastic Gradient Descentcited by 2
Built on by
Not cited yet.