# TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes

arXiv: https://arxiv.org/abs/2605.21724 · HTML: https://mlsanity.com/paper/2605.21724/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 20 May 2026.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training](https://mlsanity.com/paper/2305.14342/) (2023)
- [mHC: Manifold-Constrained Hyper-Connections](https://mlsanity.com/paper/2512.24880/) (2025)
- [Hyper-Connections](https://mlsanity.com/paper/2409.19606/) (2024)
- [mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations](https://mlsanity.com/paper/2601.05732/) (2026)
- [KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices](https://mlsanity.com/paper/2601.21579/) (2026)
- [go-mHC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices](https://mlsanity.com/paper/2604.02309/) (2026)
- [On the α-lazy version of Markov chains in estimation and testing problems](https://mlsanity.com/paper/2105.09536/) (2021)

## Built on by (most cited first, 3 of 3)

- [Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning](https://mlsanity.com/paper/2607.18130/) (2026)
- [oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions](https://mlsanity.com/paper/2609.02672/) (2026)
- [The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction](https://mlsanity.com/paper/2610.06653/) (2026)

## Abstract (arXiv)

Hyper-Connections (HC) improve residual networks by introducing learnable mixing across multiple residual streams, but unconstrained mixing leads to training instability. Manifold-Constrained Hyper-Connections (mHC) address this by enforcing approximate double stochasticity via Sinkhorn normalization, while mHC-lite ensures exact constraints through convex combinations of permutation matrices at the cost of factorial complexity. KromHC reduces this cost using Kronecker-product parameterizations, but restricts the mixing matrices to a structured submanifold of the Birkhoff polytope . We propose Transportation Birkhoff Polytope (TBP) parameterizations and their Recursive variants (RTBP), which construct exactly doubly stochastic mixing matrices with (n-1)2 degrees of freedom. Our approach avoids iterative normalization and combinatorial explosion while preserving full expressivity of the Birkhoff polytope. Empirical results on language model pre-training' demonstrate competitive performance with improved stability and scalability.