# Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics

arXiv: https://arxiv.org/abs/2606.09744 · HTML: https://mlsanity.com/paper/2606.09744/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 5 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Builds on (most cited first, 3 of 3)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Traditional and Heavy-Tailed Self Regularization in Neural Network Models](https://mlsanity.com/paper/1901.08276/) (2019)
- [The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks](https://mlsanity.com/paper/2605.06258/) (2026)

## Abstract (arXiv)

We study feed-forward ReLU networks with fixed readout and quadratic loss, and rewrite gradient descent as a collective dynamics of activation fields and conjugate fields on the training set. Working to first order in the learning rate inside a fixed activation chamber, we derive explicitly the one-, two- and three-hidden-layer cases, and then give the arbitrary-depth recursion. For one hidden layer the activation dynamics closes directly and the residual update is governed by the product of an input Gram matrix and a co-activation/backpropagation Gram matrix. For two hidden layers a conjugate field is required, but no nontrivial pullback Gram metric has yet appeared. For three hidden layers the first weight-induced pullback Gram metric enters the conjugate-field dynamics. At arbitrary depth, activation variations propagate forward through a recursive response operator (cUℓαβ), while conjugate-field variations propagate backward through an effective transport operator (cMℓαβ). Their contractions reconstruct a layerwise residual kernel [ Kαβ(L)=sumℓ=1LQαβ(ℓ-1)Sαβ(ℓ). ] The resulting description exposes a duality between push-forward and pullback transport across every layer cut, and identifies the first Gram metrics as the lowest nontrivial terms in a broader hierarchy of activation-conditioned transport operators. We deliberately stop at the level of collective fields, conjugate fields, residual kernels and cut-wise transport metrics, leaving the later tensorial geometric formulation outside the scope of this paper.