mlsanityData to 14 Oct 2026

Activation Sparsity with Weight Approximation for Faster LLM Decoding on Offloaded Weights

arXiv 2610.02598 ↗First posted 1 Oct 2026cs.LG cs.PFcited by 0

Not on the map. Why?

Abstract (arXiv)

Deploying LLMs on consumer-grade GPUs with insufficient memory to hold their weights can result in prohibitively slow inference, because decoding repeatedly transfers offloaded weights from system RAM or flash storage into GPU at much lower bandwidth than local GPU-memory access. Activation sparsity reduces these transfers by skipping weights associated with zero or near-zero activations. However, as more activation contributions are omitted, model quality eventually degrades rapidly, indicating that weights associated with small-magnitude activations collectively influence model quality sharply. In this work, we improve the trade-off between model quality and decoding performance when exploiting activation sparsity. Our key idea is to replace the binary choice of whether or not to read a weight with three options: fully retain it, approximate it using a compressed weight representation, or omit it entirely. SpAx skips weights associated with activations closest to zero, reads approximate weights for smaller-magnitude activations, and reads original weights for the largest-magnitude activations. Smaller-magnitude activations attenuate the errors introduced by approximate weights, while compressed weight representations require fewer bytes to be transferred. With weights offloaded to CPU memory, SpAx speeds up decoding by 3.86X on average (up to 5.57X) with 16-bit weights and 2.06X (up to 2.74X) with 4-bit weights, at a WikiText-2 perplexity increase of at most 10%. With weights offloaded to flash storage, the speedups are 3.31X on average (up to 4.81X) and 1.54X (up to 2.03X).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

31 of 39 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2020GLU Variants Improve Transformercited by 1,804
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  10. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  11. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  12. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
19 more
  1. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  2. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  3. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  4. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  5. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91
  6. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  7. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 45
  8. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 38
  9. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 21
  10. 2024Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Maskingcited by 11
  11. 2024CHESS: Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsificationcited by 7
  12. 2025Universal Properties of Activation Sparsity in Modern Large Language Modelscited by 7
  13. 2025La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activationcited by 6
  14. 2024Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linkingcited by 5
  15. 2025Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flashcited by 4
  16. 2025WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inferencecited by 3
  17. 2025Enabling Dynamic Sparsity in Quantized LLM Inferencecited by 1
  18. 2025VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunkingcited by 1
  19. 2026GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsity
  2. 2026To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Training
  3. 2026SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference
  4. 2023Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization
  5. 2025Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding
  6. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)
  7. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
  8. 2026The Economics of AI Decoding Chips: Rebalancing Compute, Capacity, and Bandwidth for Efficient LLM Inference