mlsanityData to 14 Oct 2026

SPARQLe: Sub-Precision Activation Representation for Quantized LLM Inference

arXiv 2606.00365 ↗First posted 29 May 2026cs.ARcited by 0

Not on the map. Why?

Abstract (arXiv)

The rapid growth in sizes of Large language models (LLMs) results in high compute and memory costs during inference. Quantization has been a significant pathway to addressing this challenge. In the quest to push the limits of quantization, weights, which are static, can often be quantized aggressively (e.g. 4 bits), while activations often require higher precision (e.g., 8 bits) to preserve accuracy, forcing hardware to operate with higher-precision datapaths. We leverage the statistical property that a significant fraction of activations are concentrated around zero, resulting in sparsity in the higher-order bits. Our proposal, SPARQLe, is a hardware-software co-design framework that exploits this sub-precision redundancy in any given quantized model. SPARQLe represents each 2k-bit activation tensor as a dense k-bit LSB tensor and a sparse k-bit MSB tensor compressed with a precision bitmap, and proposes a lightweight algorithm to increase MSB sparsity. SPARQLe reduces activation memory traffic and enables efficient computation on k-bit datapaths while preserving 2k-bit activation accuracy. SPARQLe includes an accelerator that operates directly on this hybrid format with minimal control overheads. Across the BitNet 3B, Llama2 7B, and Llama3 8B models, SPARQLe reduces prefill latency by 16-24.3% and decode latency by 13.5-23.4%, with 17-26.7% and 6.5-14.2% lower prefill and decode energy, respectively. SPARQLe demonstrates that sub-precision activation sparsity offers an effective and complementary pathway towards efficient LLM inference.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 19 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

14 of 19 references.

References · 14

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  5. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  6. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  7. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  8. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  9. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  10. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 28
  11. 2025BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inferencecited by 11
  12. 2024FlexNN: A Dataflow-aware Flexible Deep Learning Accelerator for Energy-Efficient Edge Devicescited by 9
2 more
  1. 2024TesseraQ: Ultra Low-Bit LLM Post-Training Quantization with Block Reconstructioncited by 4
  2. 2020Ax-BxP: Approximate Blocked Computation for Precision-Reconfigurable Deep Neural Network Accelerationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Enabling Dynamic Sparsity in Quantized LLM Inference
  2. 2026SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference
  3. 2025Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method
  4. 2025FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference
  5. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
  6. 2024I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models
  7. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge
  8. 2026WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsity