# Gradient Descent as Implicit EM in Distance-Based Neural Models

arXiv: https://arxiv.org/abs/2512.24780 · HTML: https://mlsanity.com/paper/2512.24780/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 31 Dec 2025.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Expectation-Maximization Attention Networks for Semantic Segmentation](https://mlsanity.com/paper/1907.13426/) (2019)
- [Neural Expectation Maximization](https://mlsanity.com/paper/1708.03498/) (2017)
- [The Bayesian Geometry of Transformer Attention](https://mlsanity.com/paper/2512.22471/) (2025)
- [Gradient Dynamics of Attention: How Cross-Entropy Sculpts Bayesian Manifolds](https://mlsanity.com/paper/2512.22473/) (2025)
- [Interpreting Neural Networks through Mahalanobis Distance](https://mlsanity.com/paper/2410.19352/) (2024)
- [Geometric Scaling of Bayesian Inference in LLMs](https://mlsanity.com/paper/2512.23752/) (2025)
- [Comparing EM with GD in Mixture Models of Two Components](https://mlsanity.com/paper/1907.03783/) (2019)
- [Neural Networks Use Distance Metrics](https://mlsanity.com/paper/2411.17932/) (2024)
- [Neural Networks Learn Distance Metrics](https://mlsanity.com/paper/2502.02103/) (2025)

## Built on by (most cited first, 1 of 1)

- [Deriving Decoder-Free Sparse Autoencoders from First Principles](https://mlsanity.com/paper/2601.06478/) (2026)

## Abstract (arXiv)

Neural networks trained with standard objectives exhibit behaviors characteristic of probabilistic inference: soft clustering, prototype specialization, and Bayesian uncertainty tracking. These phenomena appear across architectures -- in attention mechanisms, classification heads, and energy-based models -- yet existing explanations often rely on loose analogies to mixture models or post-hoc architectural interpretation. We provide a direct explanation. For any objective with log-sum-exp structure over distances or energies, the gradient with respect to each distance is exactly the negative posterior responsibility of the corresponding component: partial L / partial dj = -rj. The identity is algebraic, requiring only differentiability; it is a specialization of Fisher's identity, and its significance here is its address: standard neural objectives instantiate it without modification. The consequence is that gradient descent on such objectives performs generalized expectation-maximization implicitly, with responsibilities arising as gradients to be applied rather than auxiliary variables to be computed. This result unifies three regimes of learning: unsupervised mixture modeling, where responsibilities are fully latent; attention, where responsibilities are conditioned on queries; and cross-entropy classification, where supervision clamps responsibilities to targets. Our claims live at training time: the responsibility-weighted gradient dynamics recently documented in transformers follow from the objective's geometry. The in-context Bayesian computation that trained transformers perform at inference time is the endpoint of these dynamics, not their per-step content.