# Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning

arXiv: https://arxiv.org/abs/2604.24938 · HTML: https://mlsanity.com/paper/2604.24938/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 23 of its 25 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024)
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2304.01933/) (2023)
- [SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks](https://mlsanity.com/paper/2402.09025/) (2024)
- [EvoPress: Accurate Dynamic Model Compression via Evolutionary Search](https://mlsanity.com/paper/2410.14649/) (2024)
- [DarwinLM: Evolutionary Structured Pruning of Large Language Models](https://mlsanity.com/paper/2502.07780/) (2025)
- [Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective](https://mlsanity.com/paper/2502.14770/) (2025)
- [Prune&Comp: Free Lunch for Layer-Pruned LLMs via Iterative Pruning with Magnitude Compensation](https://mlsanity.com/paper/2507.18212/) (2025)
- [Robust Neural Pruning with Gradient Sampling Optimization for Residual Neural Networks](https://mlsanity.com/paper/2312.16020/) (2023)
- [Prompt-based Depth Pruning of Large Language Models](https://mlsanity.com/paper/2502.04348/) (2025)
- [MI-PRUN: Optimize Large Language Model Pruning via Mutual Information](https://mlsanity.com/paper/2601.07212/) (2026)
- [Towards Efficient Automatic Self-Pruning of Large Language Models](https://mlsanity.com/paper/2502.14413/) (2025)
- [LLM Compression by Block Removal with Constrained Binary Optimization](https://mlsanity.com/paper/2602.00161/) (2026)

## Built on by (most cited first, 2 of 2)

- [Does the VGGT Family Need All Its Layers?](https://mlsanity.com/paper/2609.36842/) (2026)
- [Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming](https://mlsanity.com/paper/2610.19121/) (2026)

## Abstract (arXiv)

Depth pruning improves the inference efficiency of large language models by removing Transformer blocks. Prior work typically treats layer redundancy as an inherent structural property of pretrained networks, emphasizing importance criteria and search algorithms to identify removable layers. In this study, we empirically investigate depth pruning from a functional perspective. Evaluating representative LLM families across diverse calibration configurations and multiple search algorithms, we show that different configurations produce different pruning patterns. Furthermore, under a fixed calibration configuration, complex search algorithms yield marginal performance improvements over simple one-shot methods, converging to similar pruned subsets. Overall, our results suggest that the calibration configuration plays a substantially larger role than the choice of search algorithm in shaping pruning patterns and calibration perplexity, while contributing comparably to variance in downstream reasoning accuracy. This indicates that future pruning efforts may benefit from prioritizing the calibration configuration over search complexity.