mlsanityData to 7 Oct 2026

To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Training

arXiv 2602.06183 ↗First posted 5 Feb 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Trainings of Large Language Models are generally bottlenecked by matrix multiplications. In the Transformer architecture, a large portion of these operations happens in the Feed Forward Network (FFN), and this portion increases for larger models, up to 50% of the total pretraining floating point operations. We show that we can leverage hardware-accelerated sparsity to accelerate all matrix multiplications in the FFN, with 2:4 sparsity for weights and v:n:m (Venom) sparsity for activations. Our recipe relies on sparse training steps to accelerate a large part of the pretraining, associated with regular dense training steps towards the end. Overall, models trained with this approach exhibit the same performance on our quality benchmarks, and can speed up training end-to-end by 1.4 to 1.7x. This approach is applicable to all NVIDIA GPUs starting with the A100 generation, and is orthogonal to common optimization techniques, such as, quantization, and can also be applied to mixture-of-experts model architectures.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 18 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

16 of 18 references.

References · 16

  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  3. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  4. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  5. 2021Accelerating Sparse Deep Neural Networkscited by 213
  6. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  7. 2025Pretraining Large Language Models with NVFP4cited by 67
  8. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 49
  9. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
  10. 2024Accelerating Transformer Pre-training with 2:4 Sparsitycited by 18
  11. 2023MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systemscited by 14
  12. 2023VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Corescited by 13
4 more
  1. 2025Accelerating Transformer Inference and Training with 2:4 Activation Sparsitycited by 9
  2. 2024Enabling Unstructured Sparse Acceleration on Structured Sparse Acceleratorscited by 8
  3. 2024S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-trainingcited by 7
  4. 2025Block Encoding of Sparse Matrices via Coherent Permutationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
  2. 2022S4: a High-sparsity, High-performance AI Accelerator
  3. 2024Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores
  4. 2026Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware
  5. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters
  6. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
  7. 2024Training-Free Activation Sparsity in Large Language Models
  8. 2025SparAMX: Accelerating Compressed LLMs Token Generation on AMX-powered CPUs