# Finding Experts in Transformer Models

arXiv: https://arxiv.org/abs/2005.07647 · HTML: https://mlsanity.com/paper/2005.07647/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 May 2020.

## Built on by (most cited first, 28 of 28)

- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [MoEfication: Transformer Feed-forward Layers are Mixtures of Experts](https://mlsanity.com/paper/2110.01786/) (2021)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [Neuron-level Interpretation of Deep NLP Models: A Survey](https://mlsanity.com/paper/2108.13138/) (2021)
- [Finding Skill Neurons in Pre-trained Transformer-based Language Models](https://mlsanity.com/paper/2211.07349/) (2022)
- [Discovering Latent Concepts Learned in BERT](https://mlsanity.com/paper/2205.07237/) (2022)
- [ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning](https://mlsanity.com/paper/2405.19237/) (2024)
- [Analyzing Encoded Concepts in Transformer Language Models](https://mlsanity.com/paper/2206.13289/) (2022)
- [Emergent Modularity in Pre-trained Transformers](https://mlsanity.com/paper/2305.18390/) (2023)
- [Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values](https://mlsanity.com/paper/2206.15465/) (2022)
- [Configurable Foundation Models: Building LLMs from a Modular Perspective](https://mlsanity.com/paper/2409.02877/) (2024)
- [Discovering Salient Neurons in Deep NLP Models](https://mlsanity.com/paper/2206.13288/) (2022)
- [GAM Changer: Editing Generalized Additive Models with Interactive Visualization](https://mlsanity.com/paper/2112.03245/) (2021)
- [Memorized Images in Diffusion Models share a Subspace that can be Located and Deleted](https://mlsanity.com/paper/2406.18566/) (2024)
- [NeuroX Library for Neuron Analysis of Deep NLP Models](https://mlsanity.com/paper/2305.17073/) (2023)
- [SPIN: Sparsifying and Integrating Internal Neurons in Large Language Models for Text Classification](https://mlsanity.com/paper/2311.15983/) (2023)
- [Evaluating Neuron Interpretation Methods of NLP Models](https://mlsanity.com/paper/2301.12608/) (2023)
- [RuSentEval: Linguistic Source, Encoder Force!](https://mlsanity.com/paper/2103.00573/) (2021)
- [Fine-grained Interpretation and Causation Analysis in Deep NLP Models](https://mlsanity.com/paper/2105.08039/) (2021)
- [Investigating the Encoding of Words in BERT's Neurons using Feature Textualization](https://mlsanity.com/paper/2311.08240/) (2023)
- [Towards Balancing Preference and Performance through Adaptive Personalized Explainability](https://mlsanity.com/paper/2504.13856/) (2025)
- [Continual Personalization for Diffusion Models](https://mlsanity.com/paper/2510.02296/) (2025)
- [NeST: Neuron Selective Tuning for LLM Safety](https://mlsanity.com/paper/2602.16835/) (2026)
- [Post-hoc analysis of Arabic transformer models](https://mlsanity.com/paper/2210.09990/) (2022)
- [On the Relationship between Skill Neurons and Robustness in Prompt Tuning](https://mlsanity.com/paper/2309.12263/) (2023)
- [Neuronal Group Communication for Efficient Neural representation](https://mlsanity.com/paper/2510.16851/) (2025)
- [Interpreting Arabic Transformer Models](https://mlsanity.com/paper/2201.07434/) (2022)
- [Psychological Concept Neurons: Can Neural Control Bias Probing and Shift Generation in LLMs?](https://mlsanity.com/paper/2604.11802/) (2026)

## Abstract (arXiv)

In this work we study the presence of expert units in pre-trained Transformer Models (TM), and how they impact a model's performance. We define expert units to be neurons that are able to classify a concept with a given average precision, where a concept is represented by a binary set of sentences containing the concept (or not). Leveraging the OneSec dataset (Scarlini et al., 2019), we compile a dataset of 1641 concepts that allows diverse expert units in TM to be discovered. We show that expert units are important in several ways: (1) The presence of expert units is correlated (r2=0.833) with the generalization power of TM, which allows ranking TM without requiring fine-tuning on suites of downstream tasks. We further propose an empirical method to decide how accurate such experts should be to evaluate generalization. (2) The overlap of top experts between concepts provides a sensible way to quantify concept co-learning, which can be used for explainability of unknown concepts. (3) We show how to self-condition off-the-shelf pre-trained language models to generate text with a given concept by forcing the top experts to be active, without requiring re-training the model or using additional parameters.