mlsanityData to 7 Oct 2026

Accelerating Transformer Inference and Training with 2:4 Activation Sparsity

arXiv 2503.16672 ↗First posted 20 Mar 2025cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

In this paper, we demonstrate how to leverage 2:4 sparsity, a popular hardware-accelerated GPU sparsity pattern, to activations to accelerate large language model training and inference. Crucially we exploit the intrinsic sparsity found in Squared-ReLU activations to provide this acceleration with no accuracy loss. Our approach achieves up to 1.3x faster Feed Forward Network (FFNs) in both the forwards and backwards pass. This work highlights the potential for sparsity to play a key role in accelerating large language model training and inference.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 13 of its 17 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

13 of 17 references.

References · 13

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2020GLU Variants Improve Transformercited by 1,724
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  5. 2021Accelerating Sparse Deep Neural Networkscited by 213
  6. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  7. 2024Nemotron-4 340B Technical Reportcited by 143
  8. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  9. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  10. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  11. 2024SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMscited by 18
  12. 2023Sparse Fine-tuning for Inference Acceleration of Large Language Modelscited by 14
1 more
  1. 2024Enabling Unstructured Sparse Acceleration on Structured Sparse Acceleratorscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025TorchAO: PyTorch-Native Training-to-Serving Model Optimizationcited by 25
  2. 2025DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMscited by 6
  3. 2025Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Modelscited by 1
  4. 2025Masked Gated Linear Unitcited by 0
  5. 2025Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approachescited by 0
  6. 2026To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Trainingcited by 0
  7. 2026ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsitycited by 0
  8. 2026RT-Lynx: Putting GEMM Sparsity in the Right Place for Diffusion Modelscited by 0
  9. 2026HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Modelscited by 0