Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution
Not on the map. Why?
Abstract (arXiv)
Sparse large language models (LLMs) offer an attractive direction toward efficient deployment, but adapting them to downstream tasks remains challenging. The central difficulty is to enable effective task adaptation without sacrificing the efficiency advantages of sparsity. Existing fine-tuning methods are not well-suited to this setting, as they either introduce additional dense parameters or assume a fixed sparse topology, limiting their compatibility with sparse LLMs. In this paper, we propose Sparsity Evolution Fine-Tuning (SEFT), a fine-tuning framework designed specifically for sparse LLMs. SEFT allows sparse structure to evolve during fine-tuning by periodically reallocating sparse task-specific updates and reactivating previously pruned weights when beneficial. At the same time, SEFT preserves the efficiency advantages of sparsity through topology adaptation based on parameter importance. Experiments on LLaMA, DeepSeek, and Mistral models across multiple benchmarks show that SEFT delivers stronger performance while offering superior memory and time efficiency compared to existing baselines. Our code is publicly available at: https://github.com/QiaoXiao7282/SEFT.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 57 of its 65 references that are in the map. First posted on arXiv 29 May 2025.
Builds on
57 of 65 references.
References · 57
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
38 more
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
- 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
- 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
- 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
- 2023A Comprehensive Overview of Large Language Modelscited by 648
- 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
- 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
- 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
- 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
- 2024LoRA Learns Less and Forgets Lesscited by 328
- 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
- 2023A Survey on Model Compression for Large Language Modelscited by 269
- 2021Training Neural Networks with Fixed Sparse Maskscited by 214
- 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 213
- 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 189
- 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
- 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
- 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 146
- 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
- 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
- 2020Sparse GPU Kernels for Deep Learningcited by 141
- 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
- 2023Modular Deep Learningcited by 83
- 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 79
- 2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
- 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
- 2024AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Modelscited by 36
- 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
- 2020Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoderscited by 28
- 2024Scaling Sparse Fine-Tuning to Large Language Modelscited by 26
- 2023Bold but Cautious: Unlocking the Potential of Personalized Federated Learning through Cautiously Aggressive Collaborationcited by 23
- 2022Dynamic Sparse Network for Time Series Classification: Learning What to "see''cited by 22
- 2023Sparse is Enough in Fine-tuning Pre-trained Large Language Modelscited by 21
- 2024Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruningcited by 20
- 2023Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Trainingcited by 17
- 2024Sparse High Rank Adapterscited by 17
- 2023E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentationcited by 16
50 most cited of 57 All 57 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.