mlsanityData to 7 Oct 2026

Finding Skill Neurons in Pre-trained Transformer-based Language Models

arXiv 2211.07349 ↗First posted 14 Nov 2022cs.CL cs.LGcited by 65

Not on the map. Why?

Abstract (arXiv)

Transformer-based pre-trained language models have demonstrated superior performance on various natural language processing tasks. However, it remains unclear how the skills required to handle these tasks distribute among model parameters. In this paper, we find that after prompt tuning for specific tasks, the activations of some neurons within pre-trained Transformers are highly predictive of the task labels. We dub these neurons skill neurons and confirm they encode task-specific skills by finding that: (1) Skill neurons are crucial for handling tasks. Performances of pre-trained Transformers on a task significantly drop when corresponding skill neurons are perturbed. (2) Skill neurons are task-specific. Similar tasks tend to have similar distributions of skill neurons. Furthermore, we demonstrate the skill neurons are most likely generated in pre-training rather than fine-tuning by showing that the skill neurons found with prompt tuning are also crucial for other fine-tuning methods freezing neuron weights, such as the adapter-based tuning and BitFit. We also explore the applications of skill neurons, including accelerating Transformers with network pruning and building better transferability indicators. These findings may promote further research on understanding Transformers. The source code can be obtained from https://github.com/THU-KEG/Skill-Neuron.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 54 of its 75 references that are in the map. First posted on arXiv 14 Nov 2022.

Builds on

54 of 75 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
38 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  2. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  3. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  4. 2019Language Models as Knowledge Bases?cited by 1,922
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  8. 2019How Can We Know What Language Models Know?cited by 953
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  11. 2019Are Sixteen Heads Really Better than One?cited by 803
  12. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  13. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  14. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  15. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  16. 2021Fast Model Editing at Scalecited by 491
  17. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  18. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  19. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  20. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  21. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  22. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  23. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  24. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  25. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  26. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 209
  27. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  28. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  29. 2014Object Detectors Emerge in Deep Scene CNNscited by 178
  30. 2020Compositional Explanations of Neuronscited by 172
  31. 2015Visualizing and Understanding Recurrent Networkscited by 158
  32. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 148
  33. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  34. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  35. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 110
  36. 2018On the importance of single directions for generalizationcited by 99
  37. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  38. 2021On the Pitfalls of Analyzing Individual Neurons in Language Modelscited by 61

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 20; 2024: 18; 2025: 15; 2026: 11.

65 papers.

Sort:

By area:

  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  2. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  3. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  4. 2023Function Vectors in Large Language Modelscited by 292
  5. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  6. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  7. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  8. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  9. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  10. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  11. 2024Universal Neurons in GPT2 Language Modelscited by 100
  12. 2023Soft Merging of Experts with Adaptive Routingcited by 88
38 more
  1. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  2. 2024ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruningcited by 51
  3. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 47
  4. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 38
  5. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  6. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 34
  7. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  8. 2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
  9. 2023GradientCoin: A Peer-to-Peer Decentralized Large Language Modelscited by 23
  10. 2024An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMscited by 20
  11. 2024Neuron-Level Sequential Editing for Large Language Modelscited by 18
  12. 2024Large Language Model Bias Mitigation from the Perspective of Knowledge Editingcited by 16
  13. 2023Towards a fuller understanding of neurons with Clustered Compositional Explanationscited by 15
  14. 2024Learning sum of diverse features: computational hardness and efficient gradient-based training for ridge combinationscited by 13
  15. 2025Learning Task Representations from In-Context Learningcited by 11
  16. 2025Concept Corrector: Erase concepts on the fly for text-to-image diffusion modelscited by 11
  17. 2025From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Taskcited by 10
  18. 2025Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metriccited by 10
  19. 2023Evaluating Neuron Interpretation Methods of NLP Modelscited by 8
  20. 2023Examining the Causal Effect of First Names on Language Models: The Case of Social Commonsense Reasoningcited by 8
  21. 2023The Expressibility of Polynomial based Attention Schemecited by 7
  22. 2023Language-Specific Representation of Emotion-Concept Knowledge Causally Supports Emotion Inferencecited by 6
  23. 2023CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Modelcited by 5
  24. 2024Property Neurons in Self-Supervised Speech Transformerscited by 5
  25. 2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
  26. 2025Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearningcited by 5
  27. 2026Multilingual Safety Alignment Via Sparse Weight Editingcited by 5
  28. 2026When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformerscited by 5
  29. 2023Establishing Trustworthiness: Rethinking Tasks and Model Evaluationcited by 4
  30. 2023Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMscited by 4
  31. 2025Brain-Inspired Exploration of Functional Networks and Key Neurons in Large Language Modelscited by 4
  32. 2025Triple Phase Transitions: Understanding the Learning Dynamics of Large Language Models from a Neuroscience Perspectivecited by 4
  33. 2025Identifying Multi-modal Knowledge Neurons in Pretrained Transformers via Two-stage Filteringcited by 3
  34. 2025Continual Personalization for Diffusion Modelscited by 3
  35. 2026A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RLcited by 3
  36. 2023Knowledge Rumination for Pre-trained Language Modelscited by 2
  37. 2026Weight Patching: Toward Source-Level Mechanistic Localization in LLMscited by 2
  38. 2023On the Relationship between Skill Neurons and Robustness in Prompt Tuningcited by 1

50 most cited of 65 All 65 (JSON)