mlsanityData to 7 Oct 2026

A Proximal Operator for Inducing 2:4-Sparsity

arXiv 2501.18015 ↗First posted 29 Jan 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent hardware advancements in AI Accelerators and GPUs allow to efficiently compute sparse matrix multiplications, especially when 2 out of 4 consecutive weights are set to zero. However, this so-called 2:4 sparsity usually comes at a decreased accuracy of the model. We derive a regularizer that exploits the local correlation of features to find better sparsity masks in trained models. We minimize the regularizer jointly with a local squared loss by deriving the proximal operator for which we show that it has an efficient solution in the 2:4-sparse case. After optimizing the mask, we use maskedgradient updates to further minimize the local squared loss. We illustrate our method on toy problems and apply it to pruning entire large language models up to 70B parameters. On models up to 13B we improve over previous state of the art algorithms, whilst on 70B models we match their performance.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 26 of its 34 references that are in the map. First posted on arXiv 29 Jan 2025.

Builds on

26 of 34 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2021Convex optimizationcited by 2,479
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
14 more
  1. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  2. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  3. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  4. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  5. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  6. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  7. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  8. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 49
  9. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 48
  10. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  11. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  12. 2024Accuracy is Not All You Needcited by 27
  13. 2024ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Modelscited by 18
  14. 2024Inference Optimization of Foundation Models on AI Acceleratorscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMscited by 10
  2. 2026When LLMs get significantly worse: A statistical approach to detect model degradationscited by 5
  3. 2025UniPruning: Unifying Local Metric and Global Feedback for Scalable Sparse LLMscited by 0
  4. 2026Achieving Linear Speedup for Composite Federated Learningcited by 0