mlsanityData to 14 Oct 2026

Post-Training Statistical Calibration for Higher Activation Sparsity

arXiv 2412.07174 ↗First posted 10 Dec 2024cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

We present Statistical Calibrated Activation Pruning (SCAP), a post-training activation pruning framework that (1) generalizes sparsification by input activations of Fully-Connected layers for generic and flexible application across Transformers, and (2) features a simple Mode-Centering technique to pre-calibrate activation distributions for maximizing post-training sparsity. Our results demonstrate robust Pareto efficiency compared to prior methods, translating to a 1.5x additional LLM decoding speedup against CATS at iso model quality. SCAP effectiveness is empirically verified across a wide range of models, including recent Transformer Decoders, MoE, Mamba2, Encoding Transformer, and pre-quantized models, highlighting its practicality and scalability. The code is available at: https://github.com/IntelLabs/SCAP.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

19 of 31 references.

References · 19

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2022Efficiently Scaling Transformer Inferencecited by 539
  9. 2023The Falcon Series of Open Language Modelscited by 535
  10. 2022DeiT III: Revenge of the ViTcited by 419
  11. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  12. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
7 more
  1. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  2. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  3. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  4. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 45
  5. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  6. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 38
  7. 2024ShadowLLM: Predictor-based Contextual Sparsity for Large Language Modelscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Universal Properties of Activation Sparsity in Modern Large Language Modelscited by 7
  2. 2025SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Modelscited by 2
  3. 2025Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approachescited by 1
  4. 2026Bug or Feature2: Weight Drift, Activation Sparsity and Spikescited by 1
  5. 2025Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs
  2. 2026S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations
  3. 2024From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers
  4. 2025Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models
  5. 2025TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training
  6. 2026Pretraining Transformers with Quantized Softmax in Attention
  7. 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers
  8. 2022Unified Normalization for Accelerating and Stabilizing Transformers