mlsanityData to 7 Oct 2026

Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy

arXiv 2606.09080 ↗First posted 8 Jun 2026cs.LG cs.CLcited by 1

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Pruning has emerged as a dominant paradigm for accelerating large language model (LLM) inference, spanning a broad spectrum of methods that remove computation across tokens, layers, heads, dimensions, and attention patterns. Despite sharing the same objective, these pruning approaches induce fundamentally different execution behaviors, causing realized speedups to depend heavily on hardware and kernel implementations. Consequently, the practical acceleration benefits of different pruning families remain poorly understood. In this work, we introduce a GEMM-centric taxonomy that reorganizes existing pruning methods according to the logical M, N, and K dimensions of general matrix multiplication (GEMM). Leveraging this abstraction, we build a unified benchmarking framework that enables implementation-consistent comparison across the pruning design space and systematically characterizes the acceleration--quality Pareto frontier. Our results on Llama3.1-8B show that static depth pruning remains the strongest Pareto-optimal baseline and stays closest to its theoretical acceleration upper bound in memory-bounded scenarios. During prefill, the frontier transitions from static depth at low quality loss (0%--4%), to dynamic depth at moderate loss (5%--16%), and finally to static width pruning at higher loss levels (17%--26%). These findings establish the first unified view of the practical limits of pruning-based LLM acceleration and provide guidance for future pruning research. Code is available at https://github.com/EIT-NLP/LLM-Pruning/tree/main/PruningInferSim

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 49 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

21 of 49 references.

References · 21

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  10. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  11. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  12. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
9 more
  1. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  2. 2024What Matters in Transformers? Not All Attention is Neededcited by 83
  3. 2024BlockPruner: Fine-grained Pruning for Large Language Modelscited by 35
  4. 2026HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exitcited by 15
  5. 2026From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Modelscited by 9
  6. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 8
  7. 2026UTPTrack: Towards Simple and Unified Token Pruning for Visual Trackingcited by 6
  8. 2025Informed Routing in LLMs: Smarter Token-Level Computation for Faster Inferencecited by 4
  9. 2025ELANA: A Simple Energy and Latency Analyzer for LLMscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling
  2. 2025Pruning Laws for Large Language Models
  3. 2025SPAP: Structured Pruning via Alternating Optimization and Penalty Methods
  4. 2026Unified Static-Dynamic Pruning for Efficient LLM Inference
  5. 2026POP: Prefill-Only Pruning for Efficient Large Model Inference
  6. 2023Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models
  7. 2026Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
  8. 2026SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference