# Fluctuation-based Adaptive Structured Pruning for Large Language Models

arXiv: https://arxiv.org/abs/2312.11983 · HTML: https://mlsanity.com/paper/2312.11983/

## Key facts

- Cited by 111 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch](https://mlsanity.com/paper/2102.04010/) (2021)
- [Structured Pruning of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1512.08571/) (2015)
- [Prune Once for All: Sparse Pre-Trained Language Models](https://mlsanity.com/paper/2111.05754/) (2021)
- [UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers](https://mlsanity.com/paper/2301.13741/) (2023)
- [DropNet: Reducing Neural Network Complexity via Iterative Pruning](https://mlsanity.com/paper/2207.06646/) (2022)

## Built on by (most cited first, 30 of 111)

- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods](https://mlsanity.com/paper/2402.02834/) (2024)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [MoDeGPT: Modular Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2408.09632/) (2024)
- [Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes](https://mlsanity.com/paper/2402.05406/) (2024)
- [STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs](https://mlsanity.com/paper/2408.01803/) (2024)
- [RLRC: Reinforcement Learning-based Recovery for Compressed Vision-Language-Action Models](https://mlsanity.com/paper/2506.17639/) (2025)
- [Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward](https://mlsanity.com/paper/2402.01799/) (2024)
- [DarwinLM: Evolutionary Structured Pruning of Large Language Models](https://mlsanity.com/paper/2502.07780/) (2025)
- [SlimLLM: Accurate Structured Pruning for Large Language Models](https://mlsanity.com/paper/2505.22689/) (2025)
- [Search for Efficient Large Language Models](https://mlsanity.com/paper/2409.17372/) (2024)
- [Superficial Safety Alignment Hypothesis](https://mlsanity.com/paper/2410.10862/) (2024)
- [Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing](https://mlsanity.com/paper/2502.15618/) (2025)
- [PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference](https://mlsanity.com/paper/2509.04467/) (2025)
- [ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization](https://mlsanity.com/paper/2505.02819/) (2025)
- [MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE](https://mlsanity.com/paper/2507.00390/) (2025)
- [AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment](https://mlsanity.com/paper/2411.10606/) (2024)
- [SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling](https://mlsanity.com/paper/2506.04179/) (2025)
- [HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models](https://mlsanity.com/paper/2405.10299/) (2024)
- [Dynamic Low-Rank Sparse Adaptation for Large Language Models](https://mlsanity.com/paper/2502.14816/) (2025)
- [Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retraining](https://mlsanity.com/paper/2407.19126/) (2024)
- [Layer-wise dynamic rank for compressing large language models](https://mlsanity.com/paper/2509.25622/) (2025)
- [Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression](https://mlsanity.com/paper/2602.02848/) (2026)
- [ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation](https://mlsanity.com/paper/2607.13124/) (2026)
- [Designing Large Foundation Models for Efficient Training and Inference: A Survey](https://mlsanity.com/paper/2409.01990/) (2024)
- [Forget the Data and Fine-Tuning! Just Fold the Network to Compress](https://mlsanity.com/paper/2502.10216/) (2025)
- [Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning](https://mlsanity.com/paper/2504.11409/) (2025)
- [AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression](https://mlsanity.com/paper/2604.02119/) (2026)
- [Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient](https://mlsanity.com/paper/2406.10576/) (2024)
- [Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training](https://mlsanity.com/paper/2502.03460/) (2025)

## Abstract (arXiv)

Network Pruning is a promising way to address the huge computing resource demands of the deployment and inference of Large Language Models (LLMs). Retraining-free is important for LLMs' pruning methods. However, almost all of the existing retraining-free pruning approaches for LLMs focus on unstructured pruning, which requires specific hardware support for acceleration. In this paper, we propose a novel retraining-free structured pruning framework for LLMs, named FLAP (FLuctuation-based Adaptive Structured Pruning). It is hardware-friendly by effectively reducing storage and enhancing inference speed. For effective structured pruning of LLMs, we highlight three critical elements that demand the utmost attention: formulating structured importance metrics, adaptively searching the global compressed model, and implementing compensation mechanisms to mitigate performance loss. First, FLAP determines whether the output feature map is easily recoverable when a column of weight is removed, based on the fluctuation pruning metric. Then it standardizes the importance scores to adaptively determine the global compressed model structure. At last, FLAP adds additional bias terms to recover the output feature maps using the baseline values. We thoroughly evaluate our approach on a variety of language benchmarks. Without any retraining, our method significantly outperforms the state-of-the-art methods, including LLM-Pruner and the extension of Wanda in structured pruning. The code is released at https://github.com/CASIA-IVA-Lab/FLAP.