# Gradient-based Parameter Selection for Efficient Fine-Tuning

arXiv: https://arxiv.org/abs/2312.10136 · HTML: https://mlsanity.com/paper/2312.10136/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 103 references that are in the map.
- First posted on arXiv 15 Dec 2023.

## Builds on (most cited first, 30 of 74)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)

## Built on by (most cited first, 30 of 40)

- [MC-LLaVA: Multi-Concept Personalized Vision-Language Model](https://mlsanity.com/paper/2411.11706/) (2024)
- [5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks](https://mlsanity.com/paper/2408.08345/) (2024)
- [Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark](https://mlsanity.com/paper/2402.02242/) (2024)
- [STA-VPT: SpatioTemporally Aligned Visual Prompt Tuning](https://mlsanity.com/paper/2312.10376/) (2023)
- [Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model](https://mlsanity.com/paper/2503.04543/) (2025)
- [PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models](https://mlsanity.com/paper/2506.20629/) (2025)
- [Proactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspective](https://mlsanity.com/paper/2411.18615/) (2024)
- [MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba](https://mlsanity.com/paper/2411.03855/) (2024)
- [Temporally Compressed 3D Gaussian Splatting for Dynamic Scenes](https://mlsanity.com/paper/2412.05700/) (2024)
- [Expanding Sparse Tuning for Low Memory Usage](https://mlsanity.com/paper/2411.01800/) (2024)
- [Distribution-Aligned Decoding for Efficient LLM Task Adaptation](https://mlsanity.com/paper/2509.15888/) (2025)
- [TR-PTS: Task-Relevant Parameter and Token Selection for Efficient Tuning](https://mlsanity.com/paper/2507.22872/) (2025)
- [On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentation](https://mlsanity.com/paper/2505.22444/) (2025)
- [Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates](https://mlsanity.com/paper/2512.04844/) (2025)
- [Energy-Efficient Split Learning for Fine-Tuning Large Language Models in Edge Networks](https://mlsanity.com/paper/2412.00090/) (2024)
- [Structured Gradient Guidance for Few-Shot Adaptation in Large Language Models](https://mlsanity.com/paper/2506.00726/) (2025)
- [Hiding Images in Diffusion Models by Editing Learned Score Functions](https://mlsanity.com/paper/2503.18459/) (2025)
- [Task-Aware Parameter-Efficient Fine-Tuning of Large Pre-Trained Models at the Edge](https://mlsanity.com/paper/2504.03718/) (2025)
- [Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution](https://mlsanity.com/paper/2505.24037/) (2025)
- [RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Models](https://mlsanity.com/paper/2510.01240/) (2025)
- [Per-parameter Task Arithmetic for Unlearning in Large Language Models](https://mlsanity.com/paper/2601.22030/) (2026)
- [Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Models](https://mlsanity.com/paper/2602.04509/) (2026)
- [APLA: A Simple Adaptation Method for Vision Transformers](https://mlsanity.com/paper/2503.11335/) (2025)
- [Optimizing Specific and Shared Parameters for Efficient Parameter Tuning](https://mlsanity.com/paper/2504.03450/) (2025)
- [Sparsity Outperforms Low-Rank Projections in Few-Shot Adaptation](https://mlsanity.com/paper/2504.12436/) (2025)
- [Study of Training Dynamics for Memory-Constrained Fine-Tuning](https://mlsanity.com/paper/2510.19675/) (2025)
- [Kernelized Sparse Fine-Tuning with Bi-level Parameter Competition for Vision Models](https://mlsanity.com/paper/2510.24037/) (2025)
- [GD-FPS: Growth-Driven Feedforward Parameter Selection for Efficient Fine-Tuning](https://mlsanity.com/paper/2510.27359/) (2025)
- [Parameters as Experts: Adapting Vision Models with Dynamic Parameter Routing](https://mlsanity.com/paper/2602.06862/) (2026)
- [Rethinking Adapter Placement: A Dominant Adaptation Module Perspective](https://mlsanity.com/paper/2605.06183/) (2026)

## Abstract (arXiv)

With the growing size of pre-trained models, full fine-tuning and storing all the parameters for various downstream tasks is costly and infeasible. In this paper, we propose a new parameter-efficient fine-tuning method, Gradient-based Parameter Selection (GPS), demonstrating that only tuning a few selected parameters from the pre-trained model while keeping the remainder of the model frozen can generate similar or better performance compared with the full model fine-tuning method. Different from the existing popular and state-of-the-art parameter-efficient fine-tuning approaches, our method does not introduce any additional parameters and computational costs during both the training and inference stages. Another advantage is the model-agnostic and non-destructive property, which eliminates the need for any other design specific to a particular model. Compared with the full fine-tuning, GPS achieves 3.33% (91.78% vs. 88.45%, FGVC) and 9.61% (73.1% vs. 65.57%, VTAB) improvement of the accuracy with tuning only 0.36% parameters of the pre-trained model on average over 24 image classification tasks; it also demonstrates a significant improvement of 17% and 16.8% in mDice and mIoU, respectively, on medical image segmentation task. Moreover, GPS achieves state-of-the-art performance compared with existing PEFT methods.