mlsanityData to 14 Oct 2026

The Unreasonable Ineffectiveness of the Deeper Layers

arXiv 2403.17887 ↗First posted 26 Mar 2024cs.CL cs.LG stat.MLcited by 226

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

How is knowledge stored in an LLM's weights? We study this via layer pruning: if removing a certain layer does not affect model performance in common question-answering benchmarks, then the weights in that layer are not necessary for storing the knowledge needed to answer those questions. To find these unnecessary parameters, we identify the optimal block of layers to prune by considering similarity across layers; then, to "heal" the damage, we perform a small amount of finetuning. Surprisingly, with this method we find minimal degradation of performance until after a large fraction (up to half) of the layers are removed for some common open-weight models. From a scientific perspective, the robustness of these LLMs to the deletion of layers implies either that current pretraining methods are not properly leveraging the parameters in the deeper layers of the network or that the shallow layers play a critical role in storing knowledge. For our study, we use parameter-efficient finetuning (PEFT) methods, specifically quantization and Low Rank Adapters (QLoRA), such that each of our experiments can be performed on a single 40GB A100 GPU.

Key facts. Cited by 226 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 82 of its 96 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

82 of 96 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction21 Dec

Other references · 81

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
38 more
  1. 2018Neural Ordinary Differential Equationscited by 3,779
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2023Mistral 7Bcited by 3,164
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  13. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  14. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  15. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  16. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  17. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  18. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  19. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  20. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  21. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  22. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  23. 2019Are Sixteen Heads Really Better than One?cited by 816
  24. 2016Sequence-Level Knowledge Distillationcited by 801
  25. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  26. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  27. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  28. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  29. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  30. 2023Textbooks Are All You Needcited by 569
  31. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  32. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  33. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  34. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  35. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  36. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  37. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  38. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 48; 2025: 87; 2026: 91.

226 papers.

Sort:

By area:

  1. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  2. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  3. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  4. 2025MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigmcited by 87
  5. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  6. 2025A Survey on Latent Reasoningcited by 81
  7. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  8. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 68
  9. 2026Attention Residualscited by 65
  10. 2025The Curse of Depth in Large Language Modelscited by 62
  11. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  12. 2025Do Language Models Use Their Depth Efficiently?cited by 50
38 more
  1. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  2. 2024Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similaritycited by 36
  3. 2024AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Modelscited by 36
  4. 2024Accuracy is Not All You Needcited by 31
  5. 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 31
  6. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 30
  7. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 24
  8. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 23
  9. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 20
  10. 2024Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Mergingcited by 19
  11. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationcited by 19
  12. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  13. 2024SLMRec: Distilling Large Language Models into Small for Sequential Recommendationcited by 18
  14. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 18
  15. 2025RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformerscited by 18
  16. 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 17
  17. 2024Puzzle: Distillation-Based NAS for Inference-Optimized LLMscited by 17
  18. 2024Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levelscited by 16
  19. 2024MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Designcited by 16
  20. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extractioncited by 16
  21. 2025How Do LLMs Use Their Depth?cited by 16
  22. 2025Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoningcited by 15
  23. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  24. 2024How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regressioncited by 14
  25. 2025ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearizationcited by 13
  26. 2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
  27. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoEcited by 13
  28. 2024Persistent Topological Features in Large Language Modelscited by 12
  29. 2024AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deploymentcited by 12
  30. 20252SSP: A Two-Stage Framework for Structured Pruning of LLMscited by 12
  31. 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
  32. 2024CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Informationcited by 11
  33. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  34. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learningcited by 11
  35. 2026ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Modelscited by 11
  36. 2024Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformerscited by 10
  37. 2025Learning to Skip the Middle Layers of Transformerscited by 10
  38. 2026SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Normcited by 10

50 most cited of 226 All 226 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs
  2. 2023Just CHOP: Embarrassingly Simple LLM Compression
  3. 2026LLMs can Compress LLMs: Adaptive Pruning by Agents
  4. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models
  5. 2026Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming
  6. 2024Streamlining Redundant Layers to Compress Large Language Models
  7. 2026Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning
  8. 2024A deeper look at depth pruning of LLMs