# Multi-Gate Residuals

arXiv: https://arxiv.org/abs/2605.23259 · HTML: https://mlsanity.com/paper/2605.23259/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 21 references that are in the map.
- First posted on arXiv 22 May 2026.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [The Unreasonable Ineffectiveness of the Deeper Layers](https://mlsanity.com/paper/2403.17887/) (2024)
- [Primer: Searching for Efficient Transformers for Language Modeling](https://mlsanity.com/paper/2109.08668/) (2021)
- [The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm](https://mlsanity.com/paper/2505.16932/) (2025)
- [mHC: Manifold-Constrained Hyper-Connections](https://mlsanity.com/paper/2512.24880/) (2025)
- [Hyper-Connections](https://mlsanity.com/paper/2409.19606/) (2024)
- [Attention Residuals](https://mlsanity.com/paper/2603.15031/) (2026)
- [The Curse of Depth in Large Language Models](https://mlsanity.com/paper/2502.05795/) (2025)
- [mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations](https://mlsanity.com/paper/2601.05732/) (2026)
- [SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm](https://mlsanity.com/paper/2602.08064/) (2026)
- [Mixture-of-Depths Attention](https://mlsanity.com/paper/2603.15619/) (2026)

## Built on by (most cited first, 1 of 1)

- [Multi-Stream Attention Residuals](https://mlsanity.com/paper/2610.13630/) (2026)

## Abstract (arXiv)

While Attention Residuals has shown some effectiveness in addressing the widespread issue of unbounded activation growth across deep residual layers, it inevitably incurs significant communication overhead. To circumvent this bottleneck, we propose Multi-Gate Residuals (MGR), which stabilizes activation scales without additional communication burden. It utilizes a straightforward scoring and gating mechanism to maintain multi-stream context, coupled with Attention Pooling to extract hidden states from the stream states. Empirical experiments demonstrate that MGR is practical for large-scale training and deployment, offering tangible performance improvements over existing architectures.