# mHC: Manifold-Constrained Hyper-Connections

arXiv: https://arxiv.org/abs/2512.24880 · HTML: https://mlsanity.com/paper/2512.24880/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 44 references that are in the map.
- First posted on arXiv 31 Dec 2025.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)
- [Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts](https://mlsanity.com/paper/2408.15664/) (2024)
- [Training Very Deep Networks](https://mlsanity.com/paper/1507.06228/) (2015)
- [Hyper-Connections](https://mlsanity.com/paper/2409.19606/) (2024)

## Built on by (most cited first, 30 of 88)

- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Attention Residuals](https://mlsanity.com/paper/2603.15031/) (2026)
- [On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability](https://mlsanity.com/paper/2608.30320/) (2026)
- [KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices](https://mlsanity.com/paper/2601.21579/) (2026)
- [SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm](https://mlsanity.com/paper/2602.08064/) (2026)
- [Deep Delta Learning](https://mlsanity.com/paper/2601.00417/) (2026)
- [Mixture-of-Depths Attention](https://mlsanity.com/paper/2603.15619/) (2026)
- [ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling](https://mlsanity.com/paper/2602.09009/) (2026)
- [JPmHC Dynamical Isometry via Orthogonal Hyper-Connections](https://mlsanity.com/paper/2602.18308/) (2026)
- [Orthogonal Model Merging](https://mlsanity.com/paper/2602.05943/) (2026)
- [Spectral-Sphere-Constrained Hyper-Connections](https://mlsanity.com/paper/2603.20896/) (2026)
- [Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2605.12374/) (2026)
- [Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation](https://mlsanity.com/paper/2606.03483/) (2026)
- [xHC: Expanded Hyper-Connections](https://mlsanity.com/paper/2607.14530/) (2026)
- [Don't be so Stief! Learning KV Cache low-rank approximation over the Stiefel manifold](https://mlsanity.com/paper/2601.21686/) (2026)
- [TextResNet: Decoupling and Routing Optimization Signals in Compound AI Systems via Deep Residual Tuning](https://mlsanity.com/paper/2602.08306/) (2026)
- [Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesis](https://mlsanity.com/paper/2603.10846/) (2026)
- [A Unified Foundation Model for All-in-One Multi-Modal Remote Sensing Image Restoration and Fusion with Language Prompting](https://mlsanity.com/paper/2604.05629/) (2026)
- [Optimistic Dual Averaging Unifies Modern Optimizers](https://mlsanity.com/paper/2605.11172/) (2026)
- [TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes](https://mlsanity.com/paper/2605.21724/) (2026)
- [Accelerating Birkhoff Projection for Manifold-Constrained Hyper-Connections](https://mlsanity.com/paper/2606.07574/) (2026)
- [ReactVLA: Fast and Lightweight Reactive Robot Manipulation via Improved Mean Flow Action Generation](https://mlsanity.com/paper/2606.14255/) (2026)
- [Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers](https://mlsanity.com/paper/2607.28611/) (2026)
- [SimpleGPT: Improving GPT via A Simple Normalization Strategy](https://mlsanity.com/paper/2602.01212/) (2026)
- [Beyond Linearity in Attention Projections: The Case for Nonlinear Queries](https://mlsanity.com/paper/2603.13381/) (2026)
- [YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference](https://mlsanity.com/paper/2604.13556/) (2026)
- [Rethinking Cross-Layer Information Routing in Diffusion Transformers](https://mlsanity.com/paper/2605.20708/) (2026)
- [Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability](https://mlsanity.com/paper/2606.28116/) (2026)
- [How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing](https://mlsanity.com/paper/2609.05309/) (2026)
- [Avoiding Premature Collapse: Adaptive Annealing for Entropy-Regularized Structural Inference](https://mlsanity.com/paper/2601.23039/) (2026)

## Abstract (arXiv)

Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.