mlsanityData to 7 Oct 2026

GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs

arXiv 2408.15300 ↗First posted 27 Aug 2024cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and democratized the usage of Large Language Models (LLMs). Recent studies have shown that a small subset of weights significantly impacts performance. Based on this observation, we introduce a novel PEFT method, called Gaussian noise Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only salient columns, while injecting Gaussian noise into non-salient ones. To identify these columns, we developeda generalized sensitivity metric that extends and unifies metrics from previous studies. Experiments with LLaMA models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT methods under the same computational budget. Moreover, GIFT-SW offers practical advantages to recover performance of models subjected to mixed-precision quantization with keeping salient weights in full precision.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 29 of its 60 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

29 of 60 references.

References · 29

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
17 more
  1. 2023The Falcon Series of Open Language Modelscited by 526
  2. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  3. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  4. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  5. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  6. 2017Towards Accurate Binary Convolutional Neural Networkcited by 148
  7. 2024Universal Neurons in GPT2 Language Modelscited by 100
  8. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 78
  9. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  10. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 44
  11. 2019One Epoch Is All You Needcited by 41
  12. 2020Explicit Regularisation in Gaussian Noise Injectionscited by 37
  13. 2022NIPQ: Noise proxy-based Integrated Pseudo-Quantizationcited by 35
  14. 2014Analyzing noise in autoencoders and deep networkscited by 25
  15. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 25
  16. 2023PAC-tuning:Fine-tuning Pretrained Language Models with PAC-driven Perturbed Gradient Descentcited by 10
  17. 2022QuantNAS for super resolution: searching for efficient quantization-friendly architectures against quantization noisecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 2.

8 papers.

  1. 2025Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Modelscited by 17
  2. 2024Variational Low-Rank Adaptation Using IVONcited by 5
  3. 2025Improving LoRA with Variational Learningcited by 5
  4. 2026Tracing Persona Vectors Through LLM Pretrainingcited by 3
  5. 2025From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correctioncited by 1
  6. 2026Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuningcited by 1
  7. 2024Harnessing the Power of Noise: A Survey of Techniques and Applicationscited by 0
  8. 2025Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approachescited by 0