# NeuroX Library for Neuron Analysis of Deep NLP Models

arXiv: https://arxiv.org/abs/2305.17073 · HTML: https://mlsanity.com/paper/2305.17073/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 30 of 32)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [The What-If Tool: Interactive Probing of Machine Learning Models](https://mlsanity.com/paper/1907.04135/) (2019)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [Compositional Explanations of Neurons](https://mlsanity.com/paper/2006.14032/) (2020)
- [What do Neural Machine Translation Models Learn about Morphology?](https://mlsanity.com/paper/1704.03471/) (2017)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models](https://mlsanity.com/paper/2008.05122/) (2020)
- [What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models](https://mlsanity.com/paper/1812.09355/) (2018)
- [Identifying and Controlling Important Neurons in Neural Machine Translation](https://mlsanity.com/paper/1811.01157/) (2018)
- [Explaining Explanations: Axiomatic Feature Interactions for Deep Networks](https://mlsanity.com/paper/2002.04138/) (2020)
- [Analyzing Individual Neurons in Pre-trained Language Models](https://mlsanity.com/paper/2010.02695/) (2020)
- [The emergence of number and syntax units in LSTM language models](https://mlsanity.com/paper/1903.07435/) (2019)
- [Neuron-level Interpretation of Deep NLP Models: A Survey](https://mlsanity.com/paper/2108.13138/) (2021)
- [How Important Is a Neuron?](https://mlsanity.com/paper/1805.12233/) (2018)
- [On the Pitfalls of Analyzing Individual Neurons in Language Models](https://mlsanity.com/paper/2110.07483/) (2021)
- [Intrinsic Probing through Dimension Selection](https://mlsanity.com/paper/2010.02812/) (2020)
- [Representation of linguistic form and function in recurrent neural networks](https://mlsanity.com/paper/1602.08952/) (2016)
- [How transfer learning impacts linguistic knowledge in deep NLP models?](https://mlsanity.com/paper/2105.15179/) (2021)
- [NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks](https://mlsanity.com/paper/1812.09359/) (2018)
- [Finding Experts in Transformer Models](https://mlsanity.com/paper/2005.07647/) (2020)
- [LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models](https://mlsanity.com/paper/2204.12130/) (2022)
- [Discovering Salient Neurons in Deep NLP Models](https://mlsanity.com/paper/2206.13288/) (2022)
- [Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation](https://mlsanity.com/paper/2103.13678/) (2021)
- [Evaluating Neuron Interpretation Methods of NLP Models](https://mlsanity.com/paper/2301.12608/) (2023)
- [diagNNose: A Library for Neural Activation Analysis](https://mlsanity.com/paper/2011.06819/) (2020)

## Built on by (most cited first, 11 of 11)

- [Knowledge Circuits in Pretrained Transformers](https://mlsanity.com/paper/2405.17969/) (2024)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis](https://mlsanity.com/paper/2408.01416/) (2024)
- [Latent Concept-based Explanation of NLP Models](https://mlsanity.com/paper/2404.12545/) (2024)
- [Exploring Alignment in Shared Cross-lingual Spaces](https://mlsanity.com/paper/2405.14535/) (2024)
- [Scaling up Discovery of Latent Concepts in Deep NLP Models](https://mlsanity.com/paper/2308.10263/) (2023)
- [Interpreting the Effects of Quantization on LLMs](https://mlsanity.com/paper/2508.16785/) (2025)
- [Cross-Layer Discrete Concept Discovery for Interpreting Language Models](https://mlsanity.com/paper/2506.20040/) (2025)
- [From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Models](https://mlsanity.com/paper/2506.01133/) (2025)
- [Vector Quantized Latent Concepts: A Scalable Alternative to Clustering-Based Concept Discovery](https://mlsanity.com/paper/2602.02726/) (2026)
- [SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding](https://mlsanity.com/paper/2603.08424/) (2026)

## Abstract (arXiv)

Neuron analysis provides insights into how knowledge is structured in representations and discovers the role of neurons in the network. In addition to developing an understanding of our models, neuron analysis enables various applications such as debiasing, domain adaptation and architectural search. We present NeuroX, a comprehensive open-source toolkit to conduct neuron analysis of natural language processing models. It implements various interpretation methods under a unified API, and provides a framework for data processing and evaluation, thus making it easier for researchers and practitioners to perform neuron analysis. The Python toolkit is available at https://www.github.com/fdalvi/NeuroX. Demo Video available at https://youtu.be/mLhs2YMx4u8.