mlsanityData to 7 Oct 2026

Training-Free Activation Sparsity in Large Language Models

arXiv 2408.14690 ↗First posted 26 Aug 2024cs.CL cs.AIcited by 51

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Activation sparsity can enable practical inference speedups in large language models (LLMs) by reducing the compute and memory-movement required for matrix multiplications during the forward pass. However, existing methods face limitations that inhibit widespread adoption. Some approaches are tailored towards older models with ReLU-based sparsity, while others require extensive continued pre-training on up to hundreds of billions of tokens. This paper describes TEAL, a simple training-free method that applies magnitude-based activation sparsity to hidden states throughout the entire model. TEAL achieves 40-50% model-wide sparsity with minimal performance degradation across Llama-2, Llama-3, and Mistral families, with sizes varying from 7B to 70B. We improve existing sparse kernels and demonstrate wall-clock decoding speed-ups of up to 1.53× and 1.8× at 40% and 50% model-wide sparsity. TEAL is compatible with weight quantization, enabling further efficiency gains.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 59 of its 71 references that are in the map. First posted on arXiv 26 Aug 2024.

Builds on

59 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models21 Feb

Other references · 58

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2016Layer Normalizationcited by 5,579
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  8. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  9. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  12. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  13. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  14. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  15. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  16. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  17. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  18. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  19. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  20. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  21. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  22. 2019Structured Pruning of Large Language Modelscited by 201
  23. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  24. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  25. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  26. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  27. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  28. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  29. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  30. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  31. 2013Deep Learning of Representations: Looking Forwardcited by 84
  32. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 70
  33. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  34. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 63
  35. 2023SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformercited by 60
  36. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 48
  37. 2020Sparse Weight Activation Trainingcited by 45
  38. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 25; 2026: 23.

51 papers.

Sort:

By area:

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  2. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 40
  3. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  4. 2025Softpick: No Attention Sink, No Massive Activations with Rectified Softmaxcited by 20
  5. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 18
  6. 2024BitNet a4.8: 4-bit Activations for 1-bit LLMscited by 16
  7. 2025BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMscited by 16
  8. 2025Accelerating Transformer Inference and Training with 2:4 Activation Sparsitycited by 9
  9. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
  10. 2025SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsitycited by 9
  11. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
  12. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltascited by 7
38 more
  1. 2025DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMscited by 6
  2. 2025CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Modelscited by 5
  3. 2025Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsitycited by 5
  4. 2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5
  5. 2025La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activationcited by 5
  6. 2025Universal Properties of Activation Sparsity in Modern Large Language Modelscited by 5
  7. 2025DeltaLLM: A Training-Free Framework Exploiting Temporal Sparsity for Efficient Edge LLM Inferencecited by 4
  8. 2025CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMscited by 4
  9. 2025Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flashcited by 3
  10. 2024Mixture of Hidden-Dimensions Transformercited by 2
  11. 2025WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inferencecited by 2
  12. 2026It Takes a MAESTRO To Prune Bad Expertscited by 2
  13. 2025GLASS: Global-Local Aggregation for Inference-time Sparsification of LLMscited by 1
  14. 2026HALO: Semantic-Aware Distributed LLM Inference in Lossy Edge Networkcited by 1
  15. 2026Robustness of Mixtures of Experts to Feature Noisecited by 1
  16. 2026ActTail: Global Activation Sparsity in Large Language Modelscited by 1
  17. 2026Bug or Feature2: Weight Drift, Activation Sparsity and Spikescited by 1
  18. 2025Masked Gated Linear Unitcited by 0
  19. 2025Stochastic activationscited by 0
  20. 2025VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunkingcited by 0
  21. 2025Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneitycited by 0
  22. 2026POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Modelscited by 0
  23. 2026WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsitycited by 0
  24. 2026SVD Contextual Sparsity Predictors for Fast LLM Inferencecited by 0
  25. 2026Sparser, Faster, Lighter Transformer Language Modelscited by 0
  26. 2026Gated Subspace Inference for Transformer Accelerationcited by 0
  27. 2026Compute Where it Counts: Self Optimizing Language Modelscited by 0
  28. 2026Towards the Connection between Activation Sparsity and Flat Minimacited by 0
  29. 2026RT-Lynx: Putting GEMM Sparsity in the Right Place for Diffusion Modelscited by 0
  30. 2026GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generationcited by 0
  31. 2026At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inferencecited by 0
  32. 2026SparSEEty: Extracting Tokens from Sparsity-Exploiting LLM Serving Systems via Deterministic Side Channelscited by 0
  33. 2026Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimizationcited by 0
  34. 2026Batch-wise Adaptive Pruning: Periodic Neuron Activation-Aware Weight Pruning for Language Reasoning Modelcited by 0
  35. 2026Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardwarecited by 0
  36. 2026LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inferencecited by 0
  37. 2026Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNscited by 0
  38. 2026Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decodingcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SPARQLe: Sub-Precision Activation Representation for Quantized LLM Inference ↗
  2. 2023Sparse Fine-tuning for Inference Acceleration of Large Language Models
  3. 2026ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
  4. 2025Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator
  5. 2024Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment
  6. 2026SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
  7. 2025The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM
  8. 2025Sparsity Forcing: Reinforcing Token Sparsity of MLLMs