# ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models

arXiv: https://arxiv.org/abs/2406.16635 · HTML: https://mlsanity.com/paper/2406.16635/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023)
- [Neural Architecture Search without Training](https://mlsanity.com/paper/2006.04647/) (2020)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [Zero-Cost Proxies for Lightweight NAS](https://mlsanity.com/paper/2101.08134/) (2021)
- [Channel Gating Neural Networks](https://mlsanity.com/paper/1805.12549/) (2018)
- [Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale](https://mlsanity.com/paper/2212.09095/) (2022)
- [BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget](https://mlsanity.com/paper/1906.04113/) (2019)
- [Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs](https://mlsanity.com/paper/2405.03103/) (2024)
- [PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions](https://mlsanity.com/paper/1504.08362/) (2015)

## Built on by (most cited first, 13 of 13)

- [TokenButler: Token Importance is Predictable](https://mlsanity.com/paper/2503.07518/) (2025)
- [SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity](https://mlsanity.com/paper/2506.16500/) (2025)
- [Universal Properties of Activation Sparsity in Modern Large Language Models](https://mlsanity.com/paper/2509.00454/) (2025)
- [Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity](https://mlsanity.com/paper/2505.14884/) (2025)
- [Post-Training Statistical Calibration for Higher Activation Sparsity](https://mlsanity.com/paper/2412.07174/) (2024)
- [Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification](https://mlsanity.com/paper/2502.09647/) (2025)
- [Prompt-based Depth Pruning of Large Language Models](https://mlsanity.com/paper/2502.04348/) (2025)
- [Activation Sparsity Opportunities for Compressing General Large Language Models](https://mlsanity.com/paper/2412.12178/) (2024)
- [GLASS: Global-Local Aggregation for Inference-time Sparsification of LLMs](https://mlsanity.com/paper/2508.14302/) (2025)
- [FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing](https://mlsanity.com/paper/2412.11494/) (2024)
- [Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs](https://mlsanity.com/paper/2606.10722/) (2026)
- [At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference](https://mlsanity.com/paper/2607.25504/) (2026)
- [Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding](https://mlsanity.com/paper/2609.33889/) (2026)

## Abstract (arXiv)

The high power consumption and latency-sensitive deployments of large language models (LLMs) have motivated efficiency techniques like quantization and sparsity. Contextual sparsity, where the sparsity pattern is input-dependent, is crucial in LLMs because the permanent removal of attention heads or neurons from LLMs can significantly degrade accuracy. Prior work has attempted to model contextual sparsity using neural networks trained to predict activation magnitudes, which can be used to dynamically prune structures with low predicted activation magnitude. In this paper, we look beyond magnitude-based pruning criteria to assess attention head and neuron importance in LLMs. We develop a novel predictor called ShadowLLM, which can shadow the LLM behavior and enforce better sparsity patterns, resulting in over 15% improvement in end-to-end accuracy compared to prior methods. In addition, ShadowLLM achieves up to a 20% speed-up over the state-of-the-art DejaVu framework. These enhancements are validated on Llama-2 and OPT models with up to 30 billion parameters. Our code is available at hrefhttps://github.com/abdelfattah-lab/shadowllm/ShadowLLM.