mlsanityData to 7 Oct 2026

MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections

arXiv 2502.12170 ↗First posted 13 Feb 2025cs.LG cs.AI cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

We propose MUltiway Dynamic Dense (MUDD) connections, a simple yet effective method to address the limitations of residual connections and enhance cross-layer information flow in Transformers. Unlike existing dense connection approaches with static and shared connection weights, MUDD generates connection weights dynamically depending on hidden states at each sequence position and for each decoupled input stream (the query, key, value or residual) of a Transformer block. MUDD connections can be seamlessly integrated into any Transformer architecture to create MUDDFormer. Extensive experiments show that MUDDFormer significantly outperforms Transformers across various model architectures and scales in language modeling, achieving the performance of Transformers trained with 1.8X-2.4X compute. Notably, MUDDPythia-2.8B matches Pythia-6.9B in pretraining ppl and downstream tasks and even rivals Pythia-12B in five-shot settings, while adding only 0.23% parameters and 0.4% computation. Code in JAX and PyTorch and pre-trained models are available at https://github.com/Caiyun-AI/MUDDFormer .

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 52 of its 62 references that are in the map. First posted on arXiv 13 Feb 2025.

Builds on

52 of 62 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  10. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  13. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  14. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  15. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  16. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  17. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  18. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  19. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  20. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  21. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  22. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  23. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  24. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  25. 2020Understanding the Difficulty of Training Transformerscited by 210
  26. 2015Training Very Deep Networkscited by 172
  27. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  28. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
  29. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  30. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  31. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  32. 2024Hyper-Connectionscited by 81
  33. 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averagingcited by 48
  34. 2024Talking Heads: Understanding Inter-layer Communication in Transformer Language Modelscited by 39
  35. 2024Selective Attention Improves Transformercited by 26
  36. 2021OmniNet: Omnidirectional Representations from Transformerscited by 23
  37. 2023Cross-Layer Retrospective Retrieving via Layer Attentioncited by 20
  38. 2024Strengthening Layer Interaction via Dynamic Layer Attentioncited by 10

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 21.

27 papers.

Sort:

By area:

  1. 2025mHC: Manifold-Constrained Hyper-Connectionscited by 76
  2. 2026Attention Residualscited by 57
  3. 2026Hyperloop Transformerscited by 11
  4. 2026KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matricescited by 10
  5. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  6. 2026Deep Delta Learningcited by 7
  7. 2026Delta Attention Residualscited by 7
  8. 2025HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Modelscited by 4
  9. 2025Enhancing Layer Attention Efficiency through Pruning Redundant Retrievalscited by 3
  10. 2025Virtual Width Networkscited by 3
  11. 2026Depth-Attention: Cross-Layer Value Mixing for Language Modelscited by 3
  12. 2026Residual Stream Duality in Modern Transformer Architecturescited by 2
15 more
  1. 2026Multi-Head Attention Residualscited by 2
  2. 2025Binary Autoencoder for Mechanistic Interpretability of Large Language Modelscited by 1
  3. 2026Improving LLM Final Representations with Inter-Layer Geometrycited by 1
  4. 2026Transformers with Selective Access to Early Representationscited by 1
  5. 2026Low-Rank Attention Residualscited by 1
  6. 2026Attention Projection Mixing with Exogenous Anchorscited by 0
  7. 2026HAARES Half-Split Residual Basis Routing for Deep Transformerscited by 0
  8. 2026Enhancing Layer Interaction Using Key-Correlated Layer Attentioncited by 0
  9. 2026Dual Attention Residualscited by 0
  10. 2026Dynamic Parameterization Is Not Dynamic Inferencecited by 0
  11. 2026Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depthcited by 0
  12. 2026WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixingcited by 0
  13. 2026NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Predictioncited by 0
  14. 2026M3D-Net: Hierarchical Coordination of Spatial Context, Feature Reuse, and Differential Attention for Mammography Classificationcited by 0
  15. 2026T-Router: Learning Thalamic Routing for Reasoning with Parameter-Efficient Reinforcement Learningcited by 0