mlsanityData to 7 Oct 2026

Marginals Before Conditionals

arXiv 2603.10074 ↗First posted 10 Mar 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We construct a minimal task that isolates conditional learning in neural networks: a surjective map with K-fold ambiguity, resolved by a selector token z, so H(A | B) = log K while H(A | B, z) = 0. The model learns the marginal P(A | B) first, producing a plateau at exactly log K, before acquiring the full conditional in a sharp, collective transition. The plateau has a clean decomposition: height = log K (set by ambiguity), duration = f(D) (set by dataset size D, not K). Gradient noise stabilizes the marginal solution: higher learning rates monotonically slow the transition (3.6* across a 7* η range at fixed throughput), and batch-size reduction delays escape, consistent with an entropic force opposing departure from the low-gradient marginal. Internally, a selector-routing head assembles during the plateau, leading the loss transition by ~50% of the waiting time. This is the Type 2 directional asymmetry of Papadopoulos et al. [2024], measured dynamically: we track the excess risk from log K to zero and characterize what stabilizes it, what triggers its collapse, and how long it takes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 13 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

12 of 13 references.

References · 12

  1. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  2. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  3. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  4. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  5. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  6. 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 43
  7. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  8. 2024The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and Morecited by 25
  9. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  10. 2024Arrows of Time for Large Language Modelscited by 18
  11. 2023A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descentcited by 8
  12. 2025Adaptive Heavy-Tailed Stochastic Gradient Descentcited by 2

Built on by

Not cited yet.