# Investigating Layer Importance in Large Language Models

arXiv: https://arxiv.org/abs/2409.14381 · HTML: https://mlsanity.com/paper/2409.14381/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 22 Sep 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- [Faithful to the Original: Fact Aware Neural Abstractive Summarization](https://mlsanity.com/paper/1711.04434/) (2017)
- [Future Lens: Anticipating Subsequent Tokens from a Single Hidden State](https://mlsanity.com/paper/2311.04897/) (2023)
- [Debiasing Pre-trained Contextualised Embeddings](https://mlsanity.com/paper/2101.09523/) (2021)
- [Neuron Shapley: Discovering the Responsible Neurons](https://mlsanity.com/paper/2002.09815/) (2020)

## Built on by (most cited first, 30 of 36)

- [A Survey on Latent Reasoning](https://mlsanity.com/paper/2507.06203/) (2025)
- [Understanding Layer Significance in LLM Alignment](https://mlsanity.com/paper/2410.17875/) (2024)
- [Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers](https://mlsanity.com/paper/2502.09053/) (2025)
- [Persistent Topological Features in Large Language Models](https://mlsanity.com/paper/2410.11042/) (2024)
- [LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation](https://mlsanity.com/paper/2410.13846/) (2024)
- [ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training](https://mlsanity.com/paper/2505.11739/) (2025)
- [LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions](https://mlsanity.com/paper/2505.23811/) (2025)
- [A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation](https://mlsanity.com/paper/2506.08210/) (2025)
- [Learnable Sparsity for Vision Generative Models](https://mlsanity.com/paper/2412.02852/) (2024)
- [HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMs](https://mlsanity.com/paper/2507.17394/) (2025)
- [Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models](https://mlsanity.com/paper/2505.01731/) (2025)
- [Layerwise Recall and the Geometry of Interwoven Knowledge in LLMs](https://mlsanity.com/paper/2502.10871/) (2025)
- [Statistically-Lossless Quantization of Large Language Models](https://mlsanity.com/paper/2605.02404/) (2026)
- [Explanations of Large Language Models Explain Language Representations in the Brain](https://mlsanity.com/paper/2502.14671/) (2025)
- [Rethinking Layer Relevance in Large Language Models Beyond Cosine Similarity](https://mlsanity.com/paper/2605.14075/) (2026)
- [RAP: Runtime Adaptive Pruning for LLM Inference](https://mlsanity.com/paper/2505.17138/) (2025)
- [You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations](https://mlsanity.com/paper/2511.06516/) (2025)
- [Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for Large Language Models](https://mlsanity.com/paper/2602.07804/) (2026)
- [Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scale](https://mlsanity.com/paper/2501.19099/) (2025)
- [Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models](https://mlsanity.com/paper/2602.01167/) (2026)
- [Understanding and Guiding Layer Placement in Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2602.04019/) (2026)
- [Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4](https://mlsanity.com/paper/2603.08747/) (2026)
- [RELISH: LLM REgression with a Latent Iterative State Head](https://mlsanity.com/paper/2604.01206/) (2026)
- [Feature-Guided SAE Steering for Refusal-Rate Control using Contrasting Prompts](https://mlsanity.com/paper/2511.00029/) (2025)
- [Towards Understanding Layer Contributions in Tabular In-Context Learning Models](https://mlsanity.com/paper/2511.15432/) (2025)
- [CALR: Corrective Adaptive Low-Rank Decomposition for Efficient Large Language Model Layer Compression](https://mlsanity.com/paper/2508.16680/) (2025)
- [Medical Interpretability and Knowledge Maps of Large Language Models](https://mlsanity.com/paper/2510.11390/) (2025)
- [First Attentions Last: Better Exploiting First Attentions for Efficient Transformer Training](https://mlsanity.com/paper/2510.14614/) (2025)
- [RaBiT: Residual-Aware Binarization Training for Accurate and Efficient LLMs](https://mlsanity.com/paper/2602.05367/) (2026)
- [DistillLens: Symmetric Knowledge Distillation Through Logit Lens](https://mlsanity.com/paper/2602.13567/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have gained increasing attention due to their prominent ability to understand and process texts. Nevertheless, LLMs largely remain opaque. The lack of understanding of LLMs has obstructed the deployment in safety-critical scenarios and hindered the development of better models. In this study, we advance the understanding of LLM by investigating the significance of individual layers in LLMs. We propose an efficient sampling method to faithfully evaluate the importance of layers using Shapley values, a widely used explanation framework in feature attribution and data valuation. In addition, we conduct layer ablation experiments to assess the performance degradation resulting from the exclusion of specific layers. Our findings reveal the existence of cornerstone layers, wherein certain early layers can exhibit a dominant contribution over others. Removing one cornerstone layer leads to a drastic collapse of the model performance, often reducing it to random guessing. Conversely, removing non-cornerstone layers results in only marginal performance changes. This study identifies cornerstone layers in LLMs and underscores their critical role for future research.