mlsanityData to 14 Oct 2026

SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs

arXiv 2405.16325 ↗First posted 25 May 2024cs.LG cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

We propose SLoPe, a Double-Pruned Sparse Plus Lazy Low-rank Adapter Pretraining method for LLMs that improves the accuracy of sparse LLMs while accelerating their pretraining and inference and reducing their memory footprint. Sparse pretraining of LLMs reduces the accuracy of the model, to overcome this, prior work uses dense models during fine-tuning. SLoPe improves the accuracy of sparsely pretrained models by adding low-rank adapters in the final 1% iterations of pretraining without adding significant overheads to the model pretraining and inference. In addition, SLoPe uses a double-pruned backward pass formulation that prunes the transposed weight matrix using N:M sparsity structures to enable an accelerated sparse backward pass. SLoPe accelerates the training and inference of models with billions of parameters up to 1.25× and 1.54× respectively (OPT-33B and OPT-66B) while reducing their memory usage by up to 0.63× and 0.61× for training and inference respectively.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 25 May 2024.

Builds on

42 of 59 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
30 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  5. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  6. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  7. 2019The State of Sparsity in Deep Neural Networkscited by 476
  8. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  9. 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 227
  10. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 213
  11. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  12. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  13. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
  14. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  15. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  16. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  17. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 81
  18. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 76
  19. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  20. 2023SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Modelscited by 28
  21. 2021FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attentioncited by 26
  22. 2023STEP: Learning N:M Structured Sparsity Masks from Scratch with Preconditioncited by 26
  23. 2021KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networkscited by 20
  24. 2024Accelerating Transformer Pre-training with 2:4 Sparsitycited by 19
  25. 2024Progressive Gradient Flow for Robust N:M Sparsity Training in Transformerscited by 18
  26. 2021Sparse Plus Low Rank Matrix Decomposition: A Discrete Optimization Approachcited by 14
  27. 2022Dynamic N:M Fine-grained Structured Sparse Attention Mechanismcited by 14
  28. 2023Bi-directional Masks for Efficient N:M Sparse Trainingcited by 13
  29. 2022Training Recipe for N:M Structured Sparsity with Decaying Pruning Maskcited by 11
  30. 2023MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updatescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 7.

20 papers.

Sort:

By area:

  1. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 30
  2. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
  3. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  4. 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 20
  5. 2024SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compressioncited by 12
  6. 2025Accelerating Transformer Inference and Training with 2:4 Activation Sparsitycited by 9
  7. 2025SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsitycited by 9
  8. 2024EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximationcited by 7
  9. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  10. 2025HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMscited by 3
  11. 2025PATCH: Learnable Tile-level Hybrid Sparsity for LLMscited by 3
  12. 20263BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMscited by 3
8 more
  1. 2025OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstructioncited by 2
  2. 2025Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Modelscited by 1
  3. 2026DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Trainingcited by 1
  4. 2026LEAP: Learnable End-to-End Adaptive Pruning of Large Language Modelscited by 0
  5. 2026RT-Lynx: Putting GEMM Sparsity in the Right Place for Diffusion Modelscited by 0
  6. 2026ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformerscited by 0
  7. 2026BARTER: Compact Dynamic Sparse Training with Standard Dense Kernelscited by 0
  8. 2026Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Toocited by 0