GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameters
Not on the map. Why?
Abstract (arXiv)
Sparse fine-tuning techniques adapt LLMs to downstream tasks by only tuning a sparse subset of model parameters. However, the effectiveness of sparse adaptation depends on optimally selecting the model parameters to be fine-tuned. In this work, we introduce a novel sparse fine-tuning technique named GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameters, which fine-tunes only those model parameters which have the largest gradient magnitudes on downstream tasks and the smallest pre-trained magnitudes, intuitively prioritizing parameters that are highly task-relevant, but minimally disruptive to pre-trained knowledge. Our experimentation with LLaMA3 8B and Gemma 2B as base models shows that GaLLoP consistently improves or matches the in-distribution as well as out-of-distribution performance obtained via the usage of other leading parameter-efficient fine-tuning techniques, including LoRA, DoRA, and SAFT. Our analysis demonstrates that GaLLoP mitigates catastrophic forgetting and memorization of task data, as important pre-trained parameters remain unchanged, and stabilizes performance relative to other fine-tuning techniques, robustly generalizing across most random seeds.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 47 of its 56 references that are in the map. First posted on arXiv 22 Oct 2025.
Builds on
47 of 56 references.
References · 47
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
- 2019The Curious Case of Neural Text Degenerationcited by 2,916
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
- 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
- 2018Hierarchical Neural Story Generationcited by 1,193
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
- 2021Robust fine-tuning of zero-shot modelscited by 797
- 2017Dense-Captioning Events in Videoscited by 793
35 more
- 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
- 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
- 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
- 2019Neural Text Generation with Unlikelihood Trainingcited by 503
- 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
- 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
- 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
- 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
- 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
- 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
- 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
- 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
- 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 249
- 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
- 2021Training Neural Networks with Fixed Sparse Maskscited by 214
- 2020The Hardware Lotterycited by 160
- 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
- 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
- 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
- 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
- 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
- 2023Parameter-Efficient Fine-Tuning without Introducing New Latencycited by 59
- 2025Overtrained Language Models Are Harder to Fine-Tunecited by 58
- 2020Consistency of a Recurrent Language Model With Respect to Incomplete Decodingcited by 54
- 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
- 2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
- 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 41
- 2024Scaling Sparse Fine-Tuning to Large Language Modelscited by 26
- 2023Sparse is Enough in Fine-tuning Pre-trained Large Language Modelscited by 21
- 2024LoRA Meets Dropout under a Unified Frameworkcited by 17
- 2024SAFT: Towards Out-of-Distribution Generalization in Fine-Tuningcited by 12
- 2024BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blockscited by 9
- 2021What Do You Get When You Cross Beam Search with Nucleus Sampling?cited by 7
- 2024The Unreasonable Ineffectiveness of Nucleus Sampling on Mitigating Text Memorizationcited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.