The Residual Stream's Effective Depth
Not on the map. Why?
Abstract (arXiv)
We introduce effective depth (Deff), a scalar diagnostic that treats the layer-wise residual stream of a transformer as a discrete-time process, measures how representation similarity decays with layer distance, and aggregates that profile into one number. Across sixteen decoder-only language models, Deff separates a structural consequence of residual accumulation from an empirical one: even maximally diverse orthogonal updates have the closed-form reference FL = 2L/(L+1)<2, yet fifteen of sixteen default measurements lie below FL (Qwen3.5: 32--44%, OLMo-2: 40--41%, Pythia: 23--28%). Matched references show that the gap is not caused by the persistent initial state or update-size imbalance, but is largely a calibrated signature of correlated residual updates rather than evidence that depth is unused. Symmetric position-0, token-normalisation, and top-PC controls show the regime is not reducible to BOS or top-PC artefacts: the lone above-reference default outlier joins the same regime, and all sixteen models are sub-reference after token-normalisation or top-1-PC removal. Intermediate checkpoints show that the regime is established early in OLMo-2 and stable through 5T tokens, while Pythia-1.4B follows a distinct decreasing trajectory. A controlled residual-carry intervention supports the mechanism, and Deff is best read as a global accumulated-state diagnostic, not as a capability score or pruning method.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 25 Sep 2026.
Builds on
28 of 38 references.
References · 28
- 2024The Llama 3 Herd of Modelscited by 14,486
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,761
- 2019Similarity of Neural Network Representations Revisitedcited by 1,719
- 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,323
- 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 674
- 2024OLMo: Accelerating the Science of Language Modelscited by 605
- 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 600
- 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 529
- 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 436
- 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 375
- 2018Insights on representational similarity in neural networks with canonical correlationcited by 321
- 2016On the Expressive Power of Deep Neural Networkscited by 303
16 more
- 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 301
- 2019Intrinsic dimension of data representations in deep neural networkscited by 277
- 2016Benefits of depth in neural networkscited by 250
- 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 250
- 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
- 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 221
- 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 143
- 2021Generalized Shape Metrics on Neural Representationscited by 109
- 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
- 2017Residual Connections Encourage Iterative Inferencecited by 76
- 2025The Curse of Depth in Large Language Modelscited by 59
- 2025Do Language Models Use Their Depth Efficiently?cited by 49
- 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 49
- 2024Does Representation Matter? Exploring Intermediate Layers in Large Language Modelscited by 33
- 2024Your Transformer is Secretly Linearcited by 23
- 2014On the Effective Measure of Dimension in the Analysis Cosparse Modelcited by 1
Built on by
Not cited yet.