mlsanityData to 14 Oct 2026

Residual Matrix Transformers: Scaling the Size of the Residual Stream

arXiv 2506.22696 ↗First posted 28 Jun 2025cs.LG cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

The residual stream acts as a memory bus where transformer layers both store and access features (Elhage et al., 2021). We consider changing the mechanism for retrieving and storing information in the residual stream, and replace the residual stream of the transformer with an outer product memory matrix (Kohonen, 1972, Anderson, 1972). We call this model the Residual Matrix Transformer (RMT). We find that the RMT enjoys a number of attractive properties: 1) the size of the residual stream can be scaled independently of compute and model size, improving performance, 2) the RMT can achieve the same loss as the transformer with 58% fewer FLOPS, 25% fewer parameters, and 41% fewer training tokens tokens, and 3) the RMT outperforms the transformer on downstream evaluations. We theoretically analyze the transformer and the RMT, and show that the RMT allows for more efficient scaling of the residual stream, as well as improved variance propagation properties. Code for this project can be found at https://github.com/bmac3/residual-matrix-transformer.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 28 Jun 2025.

Builds on

37 of 53 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2022Locating and Editing Factual Associations in GPTcited by 2,691
25 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  5. 2020On Layer Normalization in the Transformer Architecturecited by 922
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  8. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  9. 2023Scaling Data-Constrained Language Modelscited by 403
  10. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  11. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  12. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  13. 2019Understanding and Improving Layer Normalizationcited by 217
  14. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  15. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  16. 2021Equinox: neural networks in JAX via callable PyTrees and filtered transformationscited by 148
  17. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  18. 2018Exploiting Deep Representations for Neural Machine Translationcited by 23
  19. 2020Highway Transformer: Self-Gating Enhanced Self-Attentive Networkscited by 18
  20. 2024Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Modelscited by 18
  21. 2019Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreementcited by 11
  22. 2022Depth-Wise Attention (DWAtt): A Layer Fusion Method for Data-Efficient Classificationcited by 10
  23. 2018Dense Information Flow for Neural Machine Translationcited by 5
  24. 2020Rewiring the Transformer with Depth-Wise LSTMscited by 3
  25. 2017Improving Language Modeling using Densely Connected Recurrent Neural Networkscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 11.

12 papers.

  1. 2025mHC: Manifold-Constrained Hyper-Connectionscited by 88
  2. 2026Attention Residualscited by 65
  3. 2026mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterationscited by 17
  4. 2026KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matricescited by 11
  5. 2026Deep Delta Learningcited by 9
  6. 2026FACTUM: Mechanistic Detection of Citation Hallucination in Long-Form RAGcited by 6
  7. 2026go-mHC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matricescited by 4
  8. 2026GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimizationcited by 1
  9. 2026Learning What to Investigate Next: Meta-Reasoning for Long-Horizon Research Agentscited by 0
  10. 2026GitSwarm: Decentralized Compounding Inferencecited by 0
  11. 2026The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retractioncited by 0
  12. 2026Depth Memorycited by 0