# Task-Specific Skill Localization in Fine-tuned Language Models

arXiv: https://arxiv.org/abs/2302.06600 · HTML: https://mlsanity.com/paper/2302.06600/

## Key facts

- Cited by 105 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 61 references that are in the map.
- First posted on arXiv 13 Feb 2023.

## Builds on (most cited first, 30 of 45)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [Uniform convergence may be unable to explain generalization in deep learning](https://mlsanity.com/paper/1902.04742/) (2019)
- [Training Neural Networks with Fixed Sparse Masks](https://mlsanity.com/paper/2111.09839/) (2021)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [Surgical Fine-Tuning Improves Adaptation to Distribution Shifts](https://mlsanity.com/paper/2210.11466/) (2022)
- [SGD on Neural Networks Learns Functions of Increasing Complexity](https://mlsanity.com/paper/1905.11604/) (2019)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)

## Built on by (most cited first, 30 of 105)

- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications](https://mlsanity.com/paper/2402.05162/) (2024)
- [The Unreasonable Ineffectiveness of the Deeper Layers](https://mlsanity.com/paper/2403.17887/) (2024)
- [Mitigating the Alignment Tax of RLHF](https://mlsanity.com/paper/2309.06256/) (2023)
- [Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking](https://mlsanity.com/paper/2402.14811/) (2024)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [Localizing Task Information for Improved Model Merging and Compression](https://mlsanity.com/paper/2405.07813/) (2024)
- [Real-Time Anomaly Detection and Reactive Planning with Large Language Models](https://mlsanity.com/paper/2407.08735/) (2024)
- [LoFiT: Localized Fine-tuning on LLM Representations](https://mlsanity.com/paper/2406.01563/) (2024)
- [Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models](https://mlsanity.com/paper/2402.12048/) (2024)
- [Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic](https://mlsanity.com/paper/2408.13656/) (2024)
- [Continual Instruction Tuning for Large Multimodal Models](https://mlsanity.com/paper/2311.16206/) (2023)
- [Reinforcement Learning Finetunes Small Subnetworks in Large Language Models](https://mlsanity.com/paper/2505.11711/) (2025)
- [Interpreting and Improving Large Language Models in Arithmetic Calculation](https://mlsanity.com/paper/2409.01659/) (2024)
- [When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations](https://mlsanity.com/paper/2310.19698/) (2023)
- [Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localization](https://mlsanity.com/paper/2410.12949/) (2024)
- [The LLM Language Network: A Neuroscientific Approach for Identifying Causally Task-Relevant Units](https://mlsanity.com/paper/2411.02280/) (2024)
- [Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks](https://mlsanity.com/paper/2410.18210/) (2024)
- [MergeBench: A Benchmark for Merging Domain-Specialized LLMs](https://mlsanity.com/paper/2505.10833/) (2025)
- [A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Time](https://mlsanity.com/paper/2309.07418/) (2023)
- [Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks](https://mlsanity.com/paper/2311.09060/) (2023)
- [APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference](https://mlsanity.com/paper/2401.12200/) (2024)
- [Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity](https://mlsanity.com/paper/2406.02913/) (2024)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Mitigating Biases for Instruction-following Language Models via Bias Neurons Elimination](https://mlsanity.com/paper/2311.09627/) (2023)
- [Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging](https://mlsanity.com/paper/2410.21804/) (2024)
- [Configurable Foundation Models: Building LLMs from a Modular Perspective](https://mlsanity.com/paper/2409.02877/) (2024)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models](https://mlsanity.com/paper/2412.11041/) (2024)
- [Continual Learning via Sparse Memory Finetuning](https://mlsanity.com/paper/2510.15103/) (2025)

## Abstract (arXiv)

Pre-trained language models can be fine-tuned to solve diverse NLP tasks, including in few-shot settings. Thus fine-tuning allows the model to quickly pick up task-specific ``skills,'' but there has been limited study of where these newly-learnt skills reside inside the massive model. This paper introduces the term skill localization for this problem and proposes a solution. Given the downstream task and a model fine-tuned on that task, a simple optimization is used to identify a very small subset of parameters (~0.01% of model parameters) responsible for (>95%) of the model's performance, in the sense that grafting the fine-tuned values for just this tiny subset onto the pre-trained model gives performance almost as well as the fine-tuned model. While reminiscent of recent works on parameter-efficient fine-tuning, the novel aspects here are that: (i) No further re-training is needed on the subset (unlike, say, with lottery tickets). (ii) Notable improvements are seen over vanilla fine-tuning with respect to calibration of predictions in-distribution (40-90% error reduction) as well as the quality of predictions out-of-distribution (OOD). In models trained on multiple tasks, a stronger notion of skill localization is observed, where the sparse regions corresponding to different tasks are almost disjoint, and their overlap (when it happens) is a proxy for task similarity. Experiments suggest that localization via grafting can assist certain forms of continual learning.