# Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation

arXiv: https://arxiv.org/abs/2404.01365 · HTML: https://mlsanity.com/paper/2404.01365/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 45 of its 59 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time](https://mlsanity.com/paper/2310.17157/) (2023)

## Built on by (most cited first, 21 of 21)

- [ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference](https://mlsanity.com/paper/2410.21465/) (2024)
- [Configurable Foundation Models: Building LLMs from a Modular Perspective](https://mlsanity.com/paper/2409.02877/) (2024)
- [AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lens](https://mlsanity.com/paper/2406.03805/) (2024)
- [Mitigating Selection Bias with Node Pruning and Auxiliary Options](https://mlsanity.com/paper/2409.18857/) (2024)
- [Scalable LLM Reasoning Acceleration with Low-rank Distillation](https://mlsanity.com/paper/2505.07861/) (2025)
- [Universal Properties of Activation Sparsity in Modern Large Language Models](https://mlsanity.com/paper/2509.00454/) (2025)
- [First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models](https://mlsanity.com/paper/2408.11393/) (2024)
- [Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training](https://mlsanity.com/paper/2502.03460/) (2025)
- [Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity](https://mlsanity.com/paper/2505.14884/) (2025)
- [PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recovery](https://mlsanity.com/paper/2502.12594/) (2025)
- [Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMs](https://mlsanity.com/paper/2502.19078/) (2025)
- [Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination Mitigation](https://mlsanity.com/paper/2603.16284/) (2026)
- [GLASS: Global-Local Aggregation for Inference-time Sparsification of LLMs](https://mlsanity.com/paper/2508.14302/) (2025)
- [Kernelized Sparse Fine-Tuning with Bi-level Parameter Competition for Vision Models](https://mlsanity.com/paper/2510.24037/) (2025)
- [Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsity](https://mlsanity.com/paper/2602.00397/) (2026)
- [MOYU: A Theoretical Study on Massive Over-activation Yielded Uplifts in LLMs](https://mlsanity.com/paper/2406.12569/) (2024)
- [Machine Learning and statistical classification of CRISPR-Cas12a diagnostic assays](https://mlsanity.com/paper/2501.04413/) (2025)
- [DART-ing Through the Drift: Dynamic Tracing of Knowledge Neurons for Adaptive Inference-Time Pruning](https://mlsanity.com/paper/2601.22632/) (2026)
- [SVD Contextual Sparsity Predictors for Fast LLM Inference](https://mlsanity.com/paper/2603.14110/) (2026)
- [Unified Static-Dynamic Pruning for Efficient LLM Inference](https://mlsanity.com/paper/2607.21985/) (2026)
- [Batch-wise Adaptive Pruning: Periodic Neuron Activation-Aware Weight Pruning for Language Reasoning Model](https://mlsanity.com/paper/2608.14003/) (2026)

## Abstract (arXiv)

With the development of transformer-based large language models (LLMs), they have been applied to many fields due to their remarkable utility, but this comes at a considerable computational cost at deployment. Fortunately, some methods such as pruning or constructing a mixture of experts (MoE) aim at exploiting sparsity in transformer feedforward (FF) blocks to gain boosts in speed and reduction in memory requirements. However, these techniques can be very costly and inflexible in practice, as they often require training or are restricted to specific types of architectures. To address this, we introduce GRIFFIN, a novel training-free and calibration-free method that selects unique FF experts at the sequence level for efficient generation across a plethora of LLMs with different non-ReLU activation functions. This is possible due to a critical observation that many trained LLMs naturally produce highly structured FF activation patterns within a sequence, which we call flocking. Despite our method's simplicity, we show with 50% of the FF parameters, GRIFFIN maintains the original model's performance with little to no degradation on a variety of classification and generation tasks, all while improving latency (e.g. 1.29× and 1.25× speed-ups in Gemma 7B and Llama 2 13B, respectively, on an NVIDIA L40). Code is available at https://github.com/hdong920/GRIFFIN.