mlsanityData to 7 Oct 2026

Hyper-Connections

arXiv 2409.19606 ↗First posted 29 Sep 2024cs.LG cs.CL cs.CVcited by 81

Not on the map. Why?

Abstract (arXiv)

We present hyper-connections, a simple yet effective method that can serve as an alternative to residual connections. This approach specifically addresses common drawbacks observed in residual connection variants, such as the seesaw effect between gradient vanishing and representation collapse. Theoretically, hyper-connections allow the network to adjust the strength of connections between features at different depths and dynamically rearrange layers. We conduct experiments focusing on the pre-training of large language models, including dense and sparse models, where hyper-connections show significant performance improvements over residual connections. Additional experiments conducted on vision tasks also demonstrate similar improvements. We anticipate that this method will be broadly applicable and beneficial across a wide range of AI problems.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 29 Sep 2024.

Builds on

24 of 37 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2016Layer Normalizationcited by 5,579
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
12 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2019Root Mean Square Layer Normalizationcited by 1,425
  4. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  5. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  6. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  7. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  8. 2020Understanding the Difficulty of Training Transformerscited by 210
  9. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  10. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  11. 2023ResiDual: Transformer with Dual Residual Connectionscited by 30
  12. 2023Alternating Updates for Efficient Transformerscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 66.

81 papers.

Sort:

By area:

  1. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  2. 2025mHC: Manifold-Constrained Hyper-Connectionscited by 76
  3. 2026Attention Residualscited by 57
  4. 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 27
  5. 2026How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Modelscited by 20
  6. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  7. 2026mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterationscited by 13
  8. 2026Hyperloop Transformerscited by 11
  9. 2026KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matricescited by 10
  10. 2025Frac-Connections: Fractional Extension of Hyper-Connectionscited by 8
  11. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  12. 2026SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Normcited by 8
38 more
  1. 2025DeepCrossAttention: Supercharging Transformer Residual Connectionscited by 7
  2. 2026Deep Delta Learningcited by 7
  3. 2026Delta Attention Residualscited by 7
  4. 2026FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Spacecited by 5
  5. 2025You Do Not Fully Utilize Transformer's Representation Capacitycited by 4
  6. 2025Auto-Compressing Networkscited by 4
  7. 2025Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Headscited by 4
  8. 2026mHC-GNN: Manifold-Constrained Hyper-Connections for Graph Neural Networkscited by 4
  9. 2026ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scalingcited by 4
  10. 2026JPmHC Dynamical Isometry via Orthogonal Hyper-Connectionscited by 4
  11. 2026Mixture-of-Depths Attentioncited by 4
  12. 2025Scale-Distribution Decoupling: Enabling Stable and Effective Training of Large Language Modelscited by 3
  13. 2025Virtual Width Networkscited by 3
  14. 2026Don't be so Stief! Learning KV Cache low-rank approximation over the Stiefel manifoldcited by 3
  15. 2026Spectral-Sphere-Constrained Hyper-Connectionscited by 3
  16. 2026go-mHC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matricescited by 3
  17. 2026CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stabilitycited by 3
  18. 2026xHC: Expanded Hyper-Connectionscited by 3
  19. 2026Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformerscited by 3
  20. 2025Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformerscited by 2
  21. 2026JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulationcited by 2
  22. 2026Beyond Linearity in Attention Projections: The Case for Nonlinear Queriescited by 2
  23. 2026Residual Stream Duality in Modern Transformer Architecturescited by 2
  24. 2026Rethinking Cross-Layer Information Routing in Diffusion Transformerscited by 2
  25. 2026TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopescited by 2
  26. 2026Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigationcited by 2
  27. 2026ReactVLA: Fast and Lightweight Reactive Robot Manipulation via Improved Mean Flow Action Generationcited by 2
  28. 2025Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modelingcited by 1
  29. 2025FlowNet: Modeling Dynamic Spatio-Temporal Systems via Flow Propagationcited by 1
  30. 2026Differentiable Logic Synthesis: Spectral Coefficient Selection via Sinkhorn-Constrained Compositioncited by 1
  31. 2026A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attentioncited by 1
  32. 2026Progressive Residual Warmup for Language Model Pretrainingcited by 1
  33. 2026Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuningcited by 1
  34. 2026FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learningcited by 1
  35. 2026SNLP: Layer-Parallel Inference via Structured Newton Correctionscited by 1
  36. 2026Transformers Provably Learn to Internalize Chain-of-Thoughtcited by 1
  37. 2026Variable-Width Transformerscited by 1
  38. 2026Low-Rank Attention Residualscited by 1

50 most cited of 81 All 81 (JSON)