# go-mHC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices

arXiv: https://arxiv.org/abs/2604.02309 · HTML: https://mlsanity.com/paper/2604.02309/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 56 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [Physics-informed neural networks with hard constraints for inverse design](https://mlsanity.com/paper/2102.04626/) (2021)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)
- [Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group](https://mlsanity.com/paper/1901.08428/) (2019)
- [Full-Capacity Unitary Recurrent Neural Networks](https://mlsanity.com/paper/1611.00035/) (2016)
- [Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](https://mlsanity.com/paper/2601.07372/) (2026)
- [Hyper-Connections](https://mlsanity.com/paper/2409.19606/) (2024)
- [Orthogonal Recurrent Neural Networks with Scaled Cayley Transform](https://mlsanity.com/paper/1707.09520/) (2017)
- [Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform](https://mlsanity.com/paper/2002.01113/) (2020)
- [DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging](https://mlsanity.com/paper/2402.02622/) (2024)
- [Recurrent Orthogonal Networks and Long-Memory Tasks](https://mlsanity.com/paper/1602.06662/) (2016)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [ResiDual: Transformer with Dual Residual Connections](https://mlsanity.com/paper/2304.14802/) (2023)
- [Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric, Topological, and Algebraic Structures](https://mlsanity.com/paper/2407.09468/) (2024)
- [Residual Stream Analysis with Multi-Layer SAEs](https://mlsanity.com/paper/2409.04185/) (2024)
- [mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations](https://mlsanity.com/paper/2601.05732/) (2026)
- [Residual Matrix Transformers: Scaling the Size of the Residual Stream](https://mlsanity.com/paper/2506.22696/) (2025)
- [KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices](https://mlsanity.com/paper/2601.21579/) (2026)
- [Frac-Connections: Fractional Extension of Hyper-Connections](https://mlsanity.com/paper/2503.14125/) (2025)
- [mHC-GNN: Manifold-Constrained Hyper-Connections for Graph Neural Networks](https://mlsanity.com/paper/2601.02451/) (2026)
- [Discovering Symmetry Groups with Flow Matching](https://mlsanity.com/paper/2512.20043/) (2025)

## Built on by (most cited first, 4 of 4)

- [TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes](https://mlsanity.com/paper/2605.21724/) (2026)
- [Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning](https://mlsanity.com/paper/2607.18130/) (2026)
- [oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions](https://mlsanity.com/paper/2609.02672/) (2026)
- [The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction](https://mlsanity.com/paper/2610.06653/) (2026)

## Abstract (arXiv)

Doubly stochastic matrices enable learned mixing across residual streams, but parameterizing the set of doubly stochastic matrices (the Birkhoff polytope) exactly and efficiently remains an open challenge. Existing exact methods scale factorially with the number of streams (d), while Kronecker-factorized approaches are efficient but expressivity-limited. We introduce a novel exact parameterization grounded in the theory of generalized orthostochastic matrices, which scales as O(d3) and exposes a single hyperparameter s which continuously interpolates between a computationally efficient boundary and the fully expressive Birkhoff polytope. Building on Manifold-Constrained Hyper-Connections (mHC), a framework for learned dynamic layer connectivity, we instantiate this parameterization in go-mHC. Our method composes naturally with Kronecker-factorized methods, substantially recovering expressivity at similar FLOP costs. Spectral analysis indicates that go-mHC fills the Birkhoff polytope far more completely than Kronecker-factorized baselines. On synthetic stream-mixing tasks, go-mHC achieves the minimum theoretical loss while converging up to 10× faster. We validate our approach in a 30M parameter GPT-style language model. The expressivity, efficiency, and exactness of go-mHC offer a practical avenue for scaling d as a new dimension of model capacity.