# Efficient Task Adaptation in Large Language Models via Selective Parameter Optimization

arXiv: https://arxiv.org/abs/2604.17051 · HTML: https://mlsanity.com/paper/2604.17051/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 44 of its 57 references that are in the map.
- First posted on arXiv 18 Apr 2026.

## Builds on (most cited first, 30 of 44)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [Full Parameter Fine-tuning for Large Language Models with Limited Resources](https://mlsanity.com/paper/2306.09782/) (2023)
- [TableBench: A Comprehensive and Complex Benchmark for Table Question Answering](https://mlsanity.com/paper/2408.09174/) (2024)
- [Language Models Meet World Models: Embodied Experiences Enhance Language Models](https://mlsanity.com/paper/2305.10626/) (2023)
- [Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models](https://mlsanity.com/paper/2310.06313/) (2023)
- [IMAGDressing-v1: Customizable Virtual Dressing](https://mlsanity.com/paper/2407.12705/) (2024)
- [LoRA-Pro: Are Low-Rank Adapters Properly Optimized?](https://mlsanity.com/paper/2407.18242/) (2024)
- [Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning](https://mlsanity.com/paper/2402.18865/) (2024)
- [Boosting Consistency in Story Visualization with Rich-Contextual Conditional Diffusion Models](https://mlsanity.com/paper/2407.02482/) (2024)
- [Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model](https://mlsanity.com/paper/2502.09533/) (2025)
- [Time-aware Multiway Adaptive Fusion Network for Temporal Knowledge Graph Question Answering](https://mlsanity.com/paper/2302.12529/) (2023)
- [Question Calibration and Multi-Hop Modeling for Temporal Question Answering](https://mlsanity.com/paper/2402.13188/) (2024)
- [DABERT: Dual Attention Enhanced BERT for Semantic Matching](https://mlsanity.com/paper/2210.03454/) (2022)
- [Dual Path Modeling for Semantic Matching by Perceiving Subtle Conflicts](https://mlsanity.com/paper/2302.12530/) (2023)
- [Improving Semantic Matching through Dependency-Enhanced Pre-trained Model with Adaptive Fusion](https://mlsanity.com/paper/2210.08471/) (2022)
- [Searching for Optimal Subword Tokenization in Cross-domain NER](https://mlsanity.com/paper/2206.03352/) (2022)
- [Robust Lottery Tickets for Pre-trained Language Models](https://mlsanity.com/paper/2211.03013/) (2022)
- [Who Stole Your Data? A Method for Detecting Unauthorized RAG Theft](https://mlsanity.com/paper/2510.07728/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated excellent performance in general language understanding, generation and other tasks. However, when fine-tuning for specific domain tasks, the general knowledge accumulated in the pre-training phase is often partially overwritten or forgotten due to parameter updates, which severely limits the generalization ability and transferability of LLMs. Traditional fine-tuning strategies mostly train on the entire parameter space, ignoring the heterogeneity of model parameters, that is, some parameters are extremely important for general tasks, while other parameters are more sensitive to specific tasks. To alleviate the above problems, this paper innovatively proposes a parameter element importance evaluation method, which divides parameters into "core parameters" and "non-core parameters" by distinguishing the importance of parameters for general language ability tasks and specific domain tasks, and fixes the core parameters during fine-tuning, and only fine-tunes the non-core parameters. Extensive experiments on scientific, medical and physical tasks using GPT-J and LLaMA-3 show that our method can mitigate catastrophic forgetting while enhancing the adaptability of the model.