mlsanityData to 7 Oct 2026

Unified Static-Dynamic Pruning for Efficient LLM Inference

arXiv 2607.21985 ↗First posted 24 Jul 2026cs.DC cs.AI cs.ARcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The increasing deployment of large language models (LLMs) has magnified the computational and memory bottlenecks of autoregressive decoding, where low compute intensity and bandwidth-bound kernels dominate inference cost. Weight pruning offers a promising remedy, but existing methods remain confined to either static pruning (SP), which permanently removes redundant weights but lacks adaptivity, or dynamic pruning (DP), which adapts to input sparsity but introduces runtime irregularity. This paper presents SPDP, a unified sparse-inference framework that integrates unstructured SP with input-adaptive DP for efficient LLM inference on GPUs. SPDP co-designs a new Tiled-Column-wise Bitmap Compressed (Tiled-CBC) format and two complementary GPU kernels: (1) a CUDA-core spMspV kernel featuring Hybrid Activation-aware Dynamic Shared-Memory Bitmap Decoding (HAD-SMBD) for fine-grained, runtime activation skipping, and (2) a Tensor-Core SpMM kernel optimized for prefill computation. This joint format-kernel design harmonizes static and dynamic sparsity, maintaining bandwidth-efficient memory access and high compute intensity under both phases of LLM inference. Comprehensive evaluations on inference-optimized GPUs demonstrate that SPDP achieves 1.24x-1.37x average speedup (up to 2.51x) over state-of-the-art sparse frameworks such as SpInfer, while matching perplexity with up to 25% higher sparsity. SPDP advances the inference efficiency-quality Pareto frontier, showing that unified static-dynamic pruning can deliver substantial throughput and performance-per-watt improvements in large-scale LLM serving.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 61 references that are in the map. First posted on arXiv 24 Jul 2026.

Builds on

34 of 61 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
21 more
  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2018CoQA: A Conversational Question Answering Challengecited by 674
  4. 2024A Survey on Large Language Models for Code Generationcited by 646
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  6. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  7. 2020Sparse GPU Kernels for Deep Learningcited by 138
  8. 2023Fluctuation-based Adaptive Structured Pruning for Large Language Modelscited by 110
  9. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  10. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  11. 2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 54
  12. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  13. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 49
  14. 2022SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learningcited by 47
  15. 2024Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generationcited by 20
  16. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 14
  17. 2024High Performance Unstructured SpMM Computation Using Tensor Corescited by 7
  18. 2025DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMscited by 6
  19. 2025Acc-SpMM: Accelerating General-purpose Sparse Matrix-Matrix Multiplication with GPU Tensor Corescited by 3
  20. 2020Adaptive SpMV/SpMSpV on GPUs for Input Vectors of Varied Sparsitycited by 2
  21. 2025ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorizationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy
  2. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
  3. 2026SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
  4. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs
  5. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
  6. 2023Efficient LLM inference solution on Intel GPU
  7. 2026Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU
  8. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference