mlsanityData to 7 Oct 2026

Q-Sparse: All Large Language Models can be Fully Sparsely-Activated

arXiv 2407.10969 ↗First posted 15 Jul 2024cs.CL cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

We introduce, Q-Sparse, a simple yet effective approach to training sparsely-activated large language models (LLMs). Q-Sparse enables full sparsity of activations in LLMs which can bring significant efficiency gains in inference. This is achieved by applying top-K sparsification to the activations and the straight-through-estimator to the training. We also introduce Block Q-Sparse for batch training and inference. The key results from this work are, (1) Q-Sparse can achieve results comparable to those of baseline LLMs while being much more efficient at inference time; (2) We present an inference-optimal scaling law for sparsely-activated LLMs; (3) Q-Sparse is effective in different settings, including training-from-scratch, continue-training of off-the-shelf LLMs, and finetuning; (4) Q-Sparse works for both full-precision and 1-bit LLMs (e.g., BitNet b1.58). Particularly, the synergy of BitNet b1.58 and Q-Sparse (can be equipped with MoE) provides the cornerstone and a clear path to revolutionize the efficiency, including cost and energy consumption, of future LLMs.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

23 of 30 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
11 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  3. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  4. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  5. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  6. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  7. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  8. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  9. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  10. 2019Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answeringcited by 58
  11. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 13; 2026: 7.

24 papers.

Sort:

By area:

  1. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  2. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 41
  3. 2025BitNet b1.58 2B4T Technical Reportcited by 30
  4. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 18
  5. 2024BitNet a4.8: 4-bit Activations for 1-bit LLMscited by 16
  6. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrationscited by 16
  7. 2025BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMscited by 16
  8. 2025La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activationcited by 5
  9. 2026CSRv2: Unlocking Ultra-Sparse Embeddingscited by 5
  10. 2024Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linkingcited by 4
  11. 2025Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flashcited by 3
  12. 2025Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsitycited by 3
12 more
  1. 2025SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Modelscited by 2
  2. 2025WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inferencecited by 2
  3. 2025Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Modelscited by 1
  4. 2025TAT-VPR: Ternary Adaptive Transformer for Dynamic and Efficient Visual Place Recognitioncited by 0
  5. 2025Generalizing Scaling Laws for Dense and Sparse Large Language Modelscited by 0
  6. 2025Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approachescited by 0
  7. 2026WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsitycited by 0
  8. 2026From Unfamiliar to Familiar: Detecting Pre-training Data via Gradient Deviations in Large Language Modelscited by 0
  9. 2026Sparser, Faster, Lighter Transformer Language Modelscited by 0
  10. 2026RT-Lynx: Putting GEMM Sparsity in the Right Place for Diffusion Modelscited by 0
  11. 2026Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardwarecited by 0
  12. 2026Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
  2. 2026SPARQLe: Sub-Precision Activation Representation for Quantized LLM Inference ↗
  3. 2023Sparse Fine-tuning for Inference Acceleration of Large Language Models
  4. 2026TopK-Guided: Adaptive, Budget-Aware Activation Sparsity for Efficient LLM Inference
  5. 2024Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment
  6. 2024A Survey on Efficient Inference for Large Language Models
  7. 2025Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator
  8. 2024Sparse Upcycling: Inference Inefficient Finetuning