# The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction

arXiv: https://arxiv.org/abs/2610.06653 · HTML: https://mlsanity.com/paper/2610.06653/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 62 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 20 of 20)

- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [mHC: Manifold-Constrained Hyper-Connections](https://mlsanity.com/paper/2512.24880/) (2025)
- [Hyper-Connections](https://mlsanity.com/paper/2409.19606/) (2024)
- [Softmax Policy Gradient Methods Can Take Exponential Time to Converge](https://mlsanity.com/paper/2102.11270/) (2021)
- [MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections](https://mlsanity.com/paper/2502.12170/) (2025)
- [mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations](https://mlsanity.com/paper/2601.05732/) (2026)
- [Residual Matrix Transformers: Scaling the Size of the Residual Stream](https://mlsanity.com/paper/2506.22696/) (2025)
- [KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices](https://mlsanity.com/paper/2601.21579/) (2026)
- [JPmHC Dynamical Isometry via Orthogonal Hyper-Connections](https://mlsanity.com/paper/2602.18308/) (2026)
- [Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections](https://mlsanity.com/paper/2603.14833/) (2026)
- [Spectral-Sphere-Constrained Hyper-Connections](https://mlsanity.com/paper/2603.20896/) (2026)
- [go-mHC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices](https://mlsanity.com/paper/2604.02309/) (2026)
- [Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation](https://mlsanity.com/paper/2606.03483/) (2026)
- [TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes](https://mlsanity.com/paper/2605.21724/) (2026)
- [Accelerating Birkhoff Projection for Manifold-Constrained Hyper-Connections](https://mlsanity.com/paper/2606.07574/) (2026)
- [Interior interpretability with attention rollout: contraction and propagation profiles in Transformers](https://mlsanity.com/paper/2607.22367/) (2026)
- [How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing](https://mlsanity.com/paper/2609.05309/) (2026)
- [Avoiding Premature Collapse: Adaptive Annealing for Entropy-Regularized Structural Inference](https://mlsanity.com/paper/2601.23039/) (2026)
- [TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing](https://mlsanity.com/paper/2608.07851/) (2026)
- [oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions](https://mlsanity.com/paper/2609.02672/) (2026)

## Abstract (arXiv)

Hyper-connections widen the residual stream of a Transformer to n parallel streams. Their manifold-constrained version (mHC) mixes the streams at each layer with a doubly stochastic matrix, which it computes by Sinkhorn normalization of exponentiated logits. We give a geometric theory of this design on the Birkhoff polytope. First, a doubly stochastic mixer splits the stream into a mean channel, on which mHC is exactly a residual network, and a difference channel, which each layer contracts by its second singular value σ2 ≤ 1 - n minij Hij. Thus the extra width is a fading memory with a horizon of 1/(1-σ2) layers, and among nonnegative mixers only the permutations do not collapse. Second, the Sinkhorn-logit map is a global chart, and its logit gradient is exactly the Fisher-Rao gradient. Thus logit gradient flow follows a squared Fisher-Rao metric, and the straight-through update is exactly entropic mirror descent. Third, under logit gradient flow the logarithm of each entry moves at a rate of at most 4n3|nabla f|_∞ varepsilon, where varepsilon is the distance to the nearest permutation. Thus gradient flow approaches and leaves the vertices only at rate 1/t, but mirror descent moves at an exponential rate. Fourth, the local convergence factor of Sinkhorn is σ22, so a fixed iteration budget limits the horizon. Experiments confirm the predicted rates.