mlsanityData to 7 Oct 2026

SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models

arXiv 2405.16057 ↗First posted 25 May 2024cs.CL cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have become pivotal in advancing the field of artificial intelligence, yet their immense sizes pose significant challenges for both fine-tuning and deployment. Current post-training pruning methods, while reducing the sizes of LLMs, often fail to maintain their original performance. To address these challenges, this paper introduces SPP, a Sparsity-Preserved Parameter-efficient fine-tuning method. Different from existing post-training pruning approaches that struggle with performance retention, SPP proposes to employ lightweight learnable column and row matrices to optimize sparse LLM weights, keeping the structure and sparsity of pruned pre-trained models intact. By element-wise multiplication and residual addition, SPP ensures the consistency of model sparsity pattern and ratio during both training and weight-merging processes. We demonstrate the effectiveness of SPP by applying it to the LLaMA and LLaMA-2 model families with recent post-training pruning methods. Our results show that SPP significantly enhances the performance of models with different sparsity patterns (i.e. unstructured and N:M sparsity), especially for those with high sparsity ratios (e.g. 75%), making it a promising solution for the efficient fine-tuning of sparse LLMs. Code will be made available at https://github.com/Lucky-Lance/SPP.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 25 May 2024.

Builds on

30 of 41 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
18 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2023PaLM 2 Technical Reportcited by 1,154
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  6. 2020Aligning AI With Shared Human Valuescited by 706
  7. 2018Rethinking the Value of Network Pruningcited by 595
  8. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  9. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  10. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  11. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  12. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  13. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  14. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 150
  15. 2020Dynamic Model Pruning with Feedbackcited by 125
  16. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  17. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  18. 2023E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsitycited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 3.

11 papers.

  1. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  2. 2024Effective Interplay between Sparsity and Quantization: From Theory to Practicecited by 16
  3. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
  4. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  5. 2026Per-parameter Task Arithmetic for Unlearning in Large Language Modelscited by 2
  6. 2026Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Modelscited by 2
  7. 2024Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanismcited by 1
  8. 2025An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruningcited by 1
  9. 2025Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolutioncited by 1
  10. 2026Routing the Lottery: Adaptive Subnetworks for Heterogeneous Datacited by 1
  11. 2025TSENOR: Highly-Efficient Algorithm for Finding Transposable N:M Sparse Maskscited by 0