# Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy

arXiv: https://arxiv.org/abs/2411.03513 · HTML: https://mlsanity.com/paper/2411.03513/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 5 Nov 2024.

## Builds on (most cited first, 17 of 17)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [The Unreasonable Ineffectiveness of the Deeper Layers](https://mlsanity.com/paper/2403.17887/) (2024)
- [LaCo: Large Language Model Pruning via Layer Collapse](https://mlsanity.com/paper/2402.11187/) (2024)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra](https://mlsanity.com/paper/2404.03647/) (2024)

## Built on by (most cited first, 9 of 9)

- [Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning](https://mlsanity.com/paper/2510.02091/) (2025)
- [More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression](https://mlsanity.com/paper/2412.12706/) (2024)
- [Progressive Binarization with Semi-Structured Pruning for LLMs](https://mlsanity.com/paper/2502.01705/) (2025)
- [ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs](https://mlsanity.com/paper/2504.13237/) (2025)
- [LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment](https://mlsanity.com/paper/2412.18135/) (2024)
- [How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs](https://mlsanity.com/paper/2606.10646/) (2026)
- [IteRABRe: Iterative Recovery-Aided Block Reduction](https://mlsanity.com/paper/2503.06291/) (2025)
- [Beyond Single-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models](https://mlsanity.com/paper/2607.18280/) (2026)
- [SARATHI: Decoupled Data-Free Probing and Adaptive Reconstruction for Structured LLM Pruning](https://mlsanity.com/paper/2610.15514/) (2026)

## Abstract (arXiv)

This paper introduces a novel model compression approach through dynamic layer-specific pruning in Large Language Models (LLMs), enhancing the traditional methodology established by SliceGPT. By transitioning from constant to dynamic slicing, our method leverages the newly proposed Layer Redundancy (LR) score, which assesses how much change each layer changes its input by measuring the cosine similarity of the input to the output of the layer. We use this score to prune parts of individual layers based on redundancy in such a way that the average pruned percentage for all layers is a fixed value. We conducted extensive experiments using models like Llama3-8B and Mistral-7B on multiple datasets, evaluating different slicing bases and percentages to determine optimal configurations that balance efficiency and performance. Our findings show that our dynamic slicing approach not only maintains but, in many cases, enhances model performance compared to the baseline established by constant slicing methods. For instance, in several settings, we see performance improvements of up to 5% over the SliceGPT baseline. Additionally, a perplexity decrease by as much as 7% was observed across multiple benchmarks, validating the effectiveness of our method. The code, model weights, and datasets are open-sourced at https://github.com/RazvanDu/DynamicSlicing.