# Enhancing Large Language Model Performance with Gradient-Based Parameter Selection

arXiv: https://arxiv.org/abs/2406.15330 · HTML: https://mlsanity.com/paper/2406.15330/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2024.

## Built on by (most cited first, 9 of 9)

- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients](https://mlsanity.com/paper/2504.10766/) (2025)
- [Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Models](https://mlsanity.com/paper/2510.09658/) (2025)
- [Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates](https://mlsanity.com/paper/2512.04844/) (2025)
- [AI Progress Should Be Measured by Capability-Per-Resource, Not Scale Alone: A Framework for Gradient-Guided Resource Allocation in LLMs](https://mlsanity.com/paper/2511.01077/) (2025)
- [Towards Identification and Intervention of Safety-Critical Parameters in Large Language Models](https://mlsanity.com/paper/2604.08297/) (2026)
- [Omni-Masked Gradient Descent: Memory-Efficient Optimization via Mask Traversal with Improved Convergence](https://mlsanity.com/paper/2603.05960/) (2026)
- [Advantageous Parameter Expansion Training Makes Better Large Language Models](https://mlsanity.com/paper/2505.24241/) (2025)
- [GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation](https://mlsanity.com/paper/2508.16191/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have revolutionized lots of fields of research. Although it is well-known that fine-tuning is essential for enhancing the capabilities of LLMs, existing research suggests that there is potential redundancy in the fine-tuning process and therefore proposes to update only a subset of parameters. However, these methods fail to leverage the task-specific information to identify important parameters during training. Based on the insight that gradients inherently contain information on task-specific data, we propose Gradient-Mask Tuning (GMT), a method that selectively updates parameters during training based on their gradient information. Specifically, we compute the absolute values of the gradients and apply masking to those with relatively smaller magnitudes. Our empirical results across various tasks demonstrate that GMT not only outperforms traditional fine-tuning methods but also elevates the upper limits of LLM performance. Further analysis indicates that GMT exhibits insensitivity to mask ratio and possesses computational efficiency comparable to vanilla SFT.