# Attention Residuals

arXiv: https://arxiv.org/abs/2603.15031 · HTML: https://mlsanity.com/paper/2603.15031/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 70 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 30 of 65)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)

## Built on by (most cited first, 30 of 65)

- [Delta Attention Residuals](https://mlsanity.com/paper/2605.18855/) (2026)
- [When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression](https://mlsanity.com/paper/2605.08234/) (2026)
- [Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients](https://mlsanity.com/paper/2606.25008/) (2026)
- [Attention Drift: What Autoregressive Speculative Decoding Models Learn](https://mlsanity.com/paper/2605.09992/) (2026)
- [Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2605.12374/) (2026)
- [PowLU: An Activation Function for Stable Pre-Training of LLMs](https://mlsanity.com/paper/2605.25704/) (2026)
- [xHC: Expanded Hyper-Connections](https://mlsanity.com/paper/2607.14530/) (2026)
- [Residual Stream Duality in Modern Transformer Architectures](https://mlsanity.com/paper/2603.16039/) (2026)
- [Symbolic-Vector Attention Fusion for Collective Intelligence](https://mlsanity.com/paper/2604.03955/) (2026)
- [ReactVLA: Fast and Lightweight Reactive Robot Manipulation via Improved Mean Flow Action Generation](https://mlsanity.com/paper/2606.14255/) (2026)
- [YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference](https://mlsanity.com/paper/2604.13556/) (2026)
- [Attention Sinks and Outliers in Attention Residuals](https://mlsanity.com/paper/2605.17887/) (2026)
- [Rethinking Cross-Layer Information Routing in Diffusion Transformers](https://mlsanity.com/paper/2605.20708/) (2026)
- [Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability](https://mlsanity.com/paper/2606.28116/) (2026)
- [Edge Cluster Expansion with Radial Rotary Attention for Interatomic Potentials](https://mlsanity.com/paper/2607.10664/) (2026)
- [SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation](https://mlsanity.com/paper/2607.21553/) (2026)
- [Multi-Head Attention Residuals](https://mlsanity.com/paper/2607.27230/) (2026)
- [Exploring the Performance Frontier of Compact Unified Image Generation Models](https://mlsanity.com/paper/2608.20334/) (2026)
- [How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents](https://mlsanity.com/paper/2609.19107/) (2026)
- [GeoAlign: Geometric Feature Realignment for MLLM Spatial Reasoning](https://mlsanity.com/paper/2604.12630/) (2026)
- [Transformers with Selective Access to Early Representations](https://mlsanity.com/paper/2605.03953/) (2026)
- [Probing Routing-Conditional Calibration in Attention-Residual Transformers](https://mlsanity.com/paper/2605.09850/) (2026)
- [Multi-Gate Residuals](https://mlsanity.com/paper/2605.23259/) (2026)
- [Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing](https://mlsanity.com/paper/2607.07953/) (2026)
- [DepthBench: Measuring How Residual Connections Enable More Computational Depth](https://mlsanity.com/paper/2609.32534/) (2026)
- [Continued AI Scaling Requires Repeated Efficiency Doublings](https://mlsanity.com/paper/2603.28507/) (2026)
- [Gradient Boosting within a Single Attention Layer](https://mlsanity.com/paper/2604.03190/) (2026)
- [XAttnRes: Cross-Stage Attention Residuals for Medical Image Segmentation](https://mlsanity.com/paper/2604.03297/) (2026)
- [CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling](https://mlsanity.com/paper/2604.04250/) (2026)
- [DALM: A Domain-Algebraic Language Model via Three-Phase Structured Generation](https://mlsanity.com/paper/2604.15593/) (2026)

## Abstract (arXiv)

Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights. To address the memory and communication overhead of attending over all preceding layer outputs for large-scale model training, we introduce Block AttnRes, which partitions layers into blocks and attends over block-level representations, reducing the memory footprint while preserving most of the gains of full AttnRes. Combined with cache-based pipeline communication and a two-phase computation strategy, Block AttnRes becomes a practical drop-in replacement for standard residual connections with minimal overhead. Scaling law experiments confirm that the improvement is consistent across model sizes, and ablations validate the benefit of content-dependent depth-wise selection. We further integrate AttnRes into the Kimi Linear architecture (48B total / 3B activated parameters) and pre-train on 1.4T tokens, where AttnRes mitigates PreNorm dilution, yielding more uniform output magnitudes and gradient distribution across depth, and improves downstream performance across all evaluated tasks.