mlsanityData to 7 Oct 2026

An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruning

arXiv 2502.11439 ↗First posted 17 Feb 2025cs.CL cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Fine-tuning is an important step in adapting foundation models such as large language models to downstream tasks. To make this step more accessible to users with limited computational budgets, it is crucial to develop fine-tuning methods that are memory and computationally efficient. Sparse Fine-tuning (SpFT) and Low-rank adaptation (LoRA) are two frameworks that have emerged for addressing this problem and have been adopted widely in practice. In this work, we develop a new SpFT framework, based on ideas from neural network pruning. At a high level, we first identify "important" neurons/nodes using feature importance metrics from network pruning (specifically, we use the structural pruning method), and then perform fine-tuning by restricting to weights involving these neurons. Experiments on common language tasks show our method improves SpFT's memory efficiency by 20-50% while matching the accuracy of state-of-the-art methods like LoRA's variants. Code available at: https://github.com/CenjhihLi/sparsityfinetuning

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 57 of its 84 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

57 of 84 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
38 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  10. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2018Neural Network Acceptability Judgmentscited by 952
  13. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  14. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  15. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  16. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  17. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  18. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  19. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  20. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  21. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  22. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  23. 2023DepGraph: Towards Any Structural Pruningcited by 202
  24. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  25. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  26. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  27. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  28. 2020Sparse GPU Kernels for Deep Learningcited by 138
  29. 2021Group Fisher Pruning for Practical Network Compressioncited by 130
  30. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  31. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  32. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  33. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 58
  34. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  35. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  36. 2020SpArch: Efficient Architecture for Sparse Matrix Multiplicationcited by 47
  37. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  38. 2024Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsitycited by 27

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuningcited by 1