mlsanityData to 7 Oct 2026

S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-training

arXiv 2409.09099 ↗First posted 13 Sep 2024cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Training deep neural networks (DNNs) is costly. Fortunately, Nvidia Ampere and Hopper GPUs can accelerate matrix multiplications twice as fast as a dense equivalent by implementing 2:4 sparsity. However, previous STE-based 2:4 pre-training methods (e.g. STE with hard-thresholding, SR-STE) suffer from optimization difficulties because of discontinuous pruning function. In this study, we comprehensively analyse the bottleneck of traditional N:M sparse training and recognize three drawbacks with discontinuity: incorrect descending direction, inability to predict the amount of descent and sparse mask oscillation. In light of this, we propose S-STE, a simple yet powerful 2:4 training method that contains two parts: to continuously project weights to be 2:4 sparse, and to rescale sparse weights with a per-tensor fixed scaling factor. Besides, we adopt minimum-variance unbiased estimation for activation gradient and FP8 quantization for whole process. Results show that our method surpasses previous 2:4 pre-training recipes and is comparable even with full parameter models. Our toolkit is available at https://github.com/huyz2023/2by4-pretrain.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 38 of its 52 references that are in the map. First posted on arXiv 13 Sep 2024.

Builds on

38 of 52 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  11. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
26 more
  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  4. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  5. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  6. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  7. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  8. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  9. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  10. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  11. 2021Accelerating Sparse Deep Neural Networkscited by 213
  12. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  13. 2021Block Pruning For Faster Transformerscited by 176
  14. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  15. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  16. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  17. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  18. 2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
  19. 2023STEP: Learning N:M Structured Sparsity Masks from Scratch with Preconditioncited by 25
  20. 2023Training and inference of large language models using 8-bit floating pointcited by 20
  21. 2024Accelerating Transformer Pre-training with 2:4 Sparsitycited by 18
  22. 2023Dynamic Sparsity Is Channel-Level Sparsity Learnercited by 17
  23. 2024Progressive Gradient Flow for Robust N:M Sparsity Training in Transformerscited by 17
  24. 2023Bi-directional Masks for Efficient N:M Sparse Trainingcited by 12
  25. 2021Towards Understanding Iterative Magnitude Pruning: Why Lottery Tickets Wincited by 11
  26. 2022Minimum Variance Unbiased N:M Sparsity for the Neural Gradientscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformerscited by 2
  2. 2025Vanishing Contributions: A Unified Framework for Smooth and Iterative Model Compressioncited by 0
  3. 2025Extreme Model Compression with Structured Sparsity at Low Precisioncited by 0
  4. 2026Learnable Permutation for Structured Sparsity on Transformer Modelscited by 0
  5. 2026To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Trainingcited by 0
  6. 2026ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsitycited by 0
  7. 2026HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spacescited by 0