mlsanityData to 14 Oct 2026

Sparse is Enough in Fine-tuning Pre-trained Large Language Models

arXiv 2312.11875 ↗First posted 19 Dec 2023cs.LG cs.AI cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

With the prevalence of pre-training-fine-tuning paradigm, how to efficiently adapt the pre-trained model to the downstream tasks has been an intriguing issue. Parameter-Efficient Fine-Tuning (PEFT) methods have been proposed for low-cost adaptation. Although PEFT has demonstrated effectiveness and been widely applied, the underlying principles are still unclear. In this paper, we adopt the PAC-Bayesian generalization error bound, viewing pre-training as a shift of prior distribution which leads to a tighter bound for generalization error. We validate this shift from the perspectives of oscillations in the loss landscape and the quasi-sparsity in gradient distribution. Based on this, we propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT), and validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning. The code is accessible at https://github.com/song-wx/SIFT/.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 24 of its 43 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

24 of 43 references.

References · 24

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
12 more
  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  2. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  3. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  4. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  5. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  6. 2014Qualitatively characterizing neural network optimization problemscited by 188
  7. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  8. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  9. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  10. 2016A Strongly Quasiconvex PAC-Bayesian Boundcited by 54
  11. 2017Exploring loss function topology with cyclical learning ratescited by 11
  12. 2020Dithered backprop: A sparse and quantized backpropagation algorithm for more efficient deep neural network trainingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 7.

21 papers.

Sort:

By area:

  1. 2024Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justificationscited by 41
  2. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  3. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  4. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  5. 2024Parameter-Efficient Fine-Tuning of State Space Modelscited by 11
  6. 2025PosterGen: Aesthetic-Aware Multi-Modal Paper-to-Poster Generation via Multi-Agent LLMscited by 11
  7. 2024Sparse Matrix in Large Language Model Fine-tuningcited by 8
  8. 2025How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradientscited by 8
  9. 2025Scalable UAV Multi-Hop Networking via Multi-Agent Reinforcement Learning with Large Language Modelscited by 4
  10. 2025Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptationcited by 4
  11. 2025Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolutioncited by 2
  12. 2026MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimizationcited by 2
9 more
  1. 2025GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameterscited by 1
  2. 2026Omni-Masked Gradient Descent: Memory-Efficient Optimization via Mask Traversal with Improved Convergencecited by 1
  3. 20265% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuningcited by 1
  4. 2026Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuningcited by 1
  5. 2024Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Modelscited by 0
  6. 2025GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptationcited by 0
  7. 2025Random Initialization of Gated Sparse Adapterscited by 0
  8. 2026FedSpy-LLM: Towards Scalable and Generalizable Data Reconstruction Attacks from Gradients on LLMscited by 0
  9. 2026One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuningcited by 0