# Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective

arXiv: https://arxiv.org/abs/2502.14770 · HTML: https://mlsanity.com/paper/2502.14770/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2025.

## Built on by (most cited first, 13 of 13)

- [Flow caching for autoregressive video generation](https://mlsanity.com/paper/2602.10825/) (2026)
- [SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models](https://mlsanity.com/paper/2603.16859/) (2026)
- [MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping](https://mlsanity.com/paper/2511.15690/) (2025)
- [The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM](https://mlsanity.com/paper/2510.01650/) (2025)
- [Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training](https://mlsanity.com/paper/2509.06518/) (2025)
- [Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning](https://mlsanity.com/paper/2604.24938/) (2026)
- [The Diminishing Returns of Early-Exit Decoding in Modern LLMs](https://mlsanity.com/paper/2603.23701/) (2026)
- [Beyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective](https://mlsanity.com/paper/2606.15161/) (2026)
- [Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal](https://mlsanity.com/paper/2603.20991/) (2026)
- [Beyond Single-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models](https://mlsanity.com/paper/2607.18280/) (2026)
- [Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization](https://mlsanity.com/paper/2607.22583/) (2026)
- [CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents](https://mlsanity.com/paper/2608.07855/) (2026)
- [Looking Inside LLMs: Small-World Connectivity as a Signature of Reasoning Performance](https://mlsanity.com/paper/2610.12304/) (2026)

## Abstract (arXiv)

In this paper, we address the challenge of determining the layer-wise sparsity rates of large language models (LLMs) through a theoretical perspective. Specifically, we identify a critical issue of ''reconstruction error explosion'' in existing LLMs sparsification methods. This refers to the cumulative effect of reconstruction errors throughout the sparsification process, where errors from earlier layers propagate and amplify in subsequent layers. As a result, the overall reconstruction error increases significantly, leading to a substantial degradation in model performance. Through theoretical analysis, we derive a simple yet effective approach to layer-wise sparsity allocation that mitigates this issue. Our method uses a monotonically increasing arithmetic progression, reducing the process of determining sparsity rates for multiple layers to the determination of a single common difference hyperparameter. Remarkably, this allows for the optimal layer-wise sparsity rates to be identified with just a few trials. Both our theoretical analysis and experimental results demonstrate that this sparsity allocation scheme is near optimal. Extensive experiments show that our method significantly improves the performance of sparse LLMs across various architectures, outperforming existing layer-wise sparsity methods. Furthermore, it enhances the performance of various compression techniques and is applicable to vision and multimodal models. Notably, our method achieves a reduction of 52.10 in perplexity for the 70% sparse LLaMA2-7B model obtained via Wanda, improves average zero-shot accuracy by 10.50%, and delivers speedups of 2.63× and 2.23× on CPU and GPU, respectively.