mlsanityData to 7 Oct 2026

mHC: Manifold-Constrained Hyper-Connections

arXiv 2512.24880 ↗First posted 31 Dec 2025cs.CL cs.AI cs.LGcited by 76

Not on the map. Why?

Abstract (arXiv)

Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 38 of its 44 references that are in the map. First posted on arXiv 31 Dec 2025.

Builds on

38 of 44 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
26 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  5. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2019Root Mean Square Layer Normalizationcited by 1,425
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  12. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  13. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  14. 2017Deep Layer Aggregationcited by 436
  15. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  16. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
  17. 2015Training Very Deep Networkscited by 172
  18. 2024Hyper-Connectionscited by 81
  19. 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averagingcited by 48
  20. 2025TileLang: A Composable Tiled Programming Model for AI Systemscited by 42
  21. 2023ResiDual: Transformer with Dual Residual Connectionscited by 30
  22. 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 27
  23. 2023Cross-Layer Retrospective Retrieving via Layer Attentioncited by 20
  24. 2020Highway Transformer: Self-Gating Enhanced Self-Attentive Networkscited by 18
  25. 2025Residual Matrix Transformers: Scaling the Size of the Residual Streamcited by 8
  26. 2025DeepCrossAttention: Supercharging Transformer Residual Connectionscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 76.

76 papers.

Sort:

By area:

  1. 2026Attention Residualscited by 57
  2. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  3. 2026KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matricescited by 10
  4. 2026SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Normcited by 8
  5. 2026Deep Delta Learningcited by 7
  6. 2026ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scalingcited by 4
  7. 2026JPmHC Dynamical Isometry via Orthogonal Hyper-Connectionscited by 4
  8. 2026Mixture-of-Depths Attentioncited by 4
  9. 2026Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Modelscited by 4
  10. 2026Don't be so Stief! Learning KV Cache low-rank approximation over the Stiefel manifoldcited by 3
  11. 2026Orthogonal Model Mergingcited by 3
  12. 2026TextResNet: Decoupling and Routing Optimization Signals in Compound AI Systems via Deep Residual Tuningcited by 3
38 more
  1. 2026Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesiscited by 3
  2. 2026Spectral-Sphere-Constrained Hyper-Connectionscited by 3
  3. 2026A Unified Foundation Model for All-in-One Multi-Modal Remote Sensing Image Restoration and Fusion with Language Promptingcited by 3
  4. 2026xHC: Expanded Hyper-Connectionscited by 3
  5. 2026Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformerscited by 3
  6. 2026SimpleGPT: Improving GPT via A Simple Normalization Strategycited by 2
  7. 2026Beyond Linearity in Attention Projections: The Case for Nonlinear Queriescited by 2
  8. 2026Optimistic Dual Averaging Unifies Modern Optimizerscited by 2
  9. 2026Rethinking Cross-Layer Information Routing in Diffusion Transformerscited by 2
  10. 2026TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopescited by 2
  11. 2026Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigationcited by 2
  12. 2026Accelerating Birkhoff Projection for Manifold-Constrained Hyper-Connectionscited by 2
  13. 2026ReactVLA: Fast and Lightweight Reactive Robot Manipulation via Improved Mean Flow Action Generationcited by 2
  14. 2026A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attentioncited by 1
  15. 2026Revisiting the Shape Convention of Transformer Language Modelscited by 1
  16. 2026Progressive Residual Warmup for Language Model Pretrainingcited by 1
  17. 2026Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuningcited by 1
  18. 2026Transformers with Selective Access to Early Representationscited by 1
  19. 2026FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learningcited by 1
  20. 2026SNLP: Layer-Parallel Inference via Structured Newton Correctionscited by 1
  21. 2026A First-Order Mean Field Control Analysis of Transformer Layers under Cross-Entropy Trainingcited by 1
  22. 2026Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instabilitycited by 1
  23. 2026Low-Rank Attention Residualscited by 1
  24. 2026Geometric and Dynamic Scaling in Deep Transformerscited by 0
  25. 2026White-Box mHC: Electromagnetic Spectrum-Aware and Interpretable Stream Interactions for Hyperspectral Image Classificationcited by 0
  26. 2026Avoiding Premature Collapse: Adaptive Annealing for Entropy-Regularized Structural Inferencecited by 0
  27. 2026mHC-HSI: Clustering-Guided Hyper-Connection Mamba for Hyperspectral Image Classificationcited by 0
  28. 2026FlatLands: Generative Floormap Completion From a Single Egocentric Viewcited by 0
  29. 2026GradAttn: Replacing Fixed Residual Connections with Task-Modulated Attention Pathwayscited by 0
  30. 2026CliffSearch: Structured Agentic Co-Evolution over Theory and Code for Scientific Algorithm Discoverycited by 0
  31. 2026YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inferencecited by 0
  32. 2026Beyond the Laplacian: Doubly Stochastic Matrices for Graph Neural Networkscited by 0
  33. 2026Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scalingcited by 0
  34. 2026Can an MLP Absorb Its Own Skip Connection Exactly?cited by 0
  35. 2026Graph Normalization: Fast Binarizing Dynamics for Differentiable MWIScited by 0
  36. 2026Cubit: Token Mixer with Kernel Ridge Regressioncited by 0
  37. 2026mHC-SSM: Manifold-Constrained Hyper-Connections for State Space Language Models with Stream-Specialized Adapterscited by 0
  38. 2026Efficient and provably convergent end-to-end training of deep neural networks with linear constraintscited by 0

50 most cited of 76 All 76 (JSON)