# Marginals Before Conditionals

arXiv: https://arxiv.org/abs/2603.10074 · HTML: https://mlsanity.com/paper/2603.10074/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 13 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Builds on (most cited first, 12 of 12)

- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit](https://mlsanity.com/paper/2207.08799/) (2022)
- [First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise](https://mlsanity.com/paper/1906.09069/) (2019)
- [Symmetry Induces Structure and Constraint of Learning](https://mlsanity.com/paper/2309.16932/) (2023)
- [The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More](https://mlsanity.com/paper/2406.05183/) (2024)
- [Arrows of Time for Large Language Models](https://mlsanity.com/paper/2401.17505/) (2024)
- [Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent](https://mlsanity.com/paper/2402.07193/) (2024)
- [A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descent](https://mlsanity.com/paper/2310.00692/) (2023)
- [Adaptive Heavy-Tailed Stochastic Gradient Descent](https://mlsanity.com/paper/2508.21353/) (2025)

## Built on by (most cited first, 1 of 1)

- [Tracking States or Tracking Cosets? An Algebraic Account of Learned State Tracking](https://mlsanity.com/paper/2609.29951/) (2026)

## Abstract (arXiv)

We construct a minimal task that isolates conditional learning in neural networks: a surjective map with K-fold ambiguity, resolved by a selector token z, so H(A | B) = log K while H(A | B, z) = 0. The model learns the marginal P(A | B) first, producing a plateau at exactly log K, before acquiring the full conditional in a sharp, collective transition. The plateau has a clean decomposition: height = log K (set by ambiguity), duration = f(D) (set by dataset size D, not K). Gradient noise stabilizes the marginal solution: higher learning rates monotonically slow the transition (3.6* across a 7* η range at fixed throughput), and batch-size reduction delays escape, consistent with an entropic force opposing departure from the low-gradient marginal. Internally, a selector-routing head assembles during the plateau, leading the loss transition by ~50% of the waiting time. This is the Type 2 directional asymmetry of Papadopoulos et al. [2024], measured dynamically: we track the excess risk from log K to zero and characterize what stabilizes it, what triggers its collapse, and how long it takes.