# What Does Layer-Importance Reveal About Transformers and State-Space Models?

arXiv: https://arxiv.org/abs/2609.16537 · HTML: https://mlsanity.com/paper/2609.16537/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 15 Sep 2026.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 20 of 20)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024)
- [The Unreasonable Ineffectiveness of the Deeper Layers](https://mlsanity.com/paper/2403.17887/) (2024)
- [Falcon Mamba: The First Competitive Attention-free 7B Language Model](https://mlsanity.com/paper/2410.05355/) (2024)
- [MambaLRP: Explaining Selective State Space Sequence Models](https://mlsanity.com/paper/2406.07592/) (2024)
- [AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality](https://mlsanity.com/paper/2410.10054/) (2024)
- [Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning](https://mlsanity.com/paper/2510.02091/) (2025)
- [PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models](https://mlsanity.com/paper/2506.20629/) (2025)
- [Layer Importance for Mathematical Reasoning is Forged in Pre-Training and Invariant after Post-Training](https://mlsanity.com/paper/2506.22638/) (2025)
- [Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models](https://mlsanity.com/paper/2501.17088/) (2025)
- [ShapLoRA: Allocation of Low-rank Adaption on Large Language Models via Shapley Value Inspired Importance Estimation](https://mlsanity.com/paper/2601.17921/) (2026)
- [TELL-TALE: Task Efficient LLMs with Task Aware Layer Elimination](https://mlsanity.com/paper/2510.22767/) (2025)

## Abstract (arXiv)

Transformers and state-space models (SSMs) are the two dominant families of sequence models, and a central open question is how far the analytical knowledge built for transformers transfers to SSMs. We address this through the lens of layer importance which underpins compression, selective fine-tuning, and interpretability across both families. We decompose layer importance into two distinct notions. Necessity captures how much the pretrained model depends on a layer's existing contribution, measured by the loss increase from bypassing it. Plasticity captures where the model absorbs new information during fine-tuning, measured by the magnitude of task-specific weight updates. Our analysis reveals that the two families behave fundamentally differently: in every evaluated residual transformer up to 14B parameters, Necessity and Plasticity anti-align across depth, whereas in the evaluated Mamba-style SSMs they point to overlapping regions. The sign of this alignment also predicts downstream adaptation behavior. In the evaluated transformers, concentrating updates in the most plastic layers increases catastrophic forgetting, while this tier-dependent effect disappears in the evaluated Mamba-style SSMs.