mlsanityData to 14 Oct 2026

NeuroX Library for Neuron Analysis of Deep NLP Models

arXiv 2305.17073 ↗First posted 26 May 2023cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Neuron analysis provides insights into how knowledge is structured in representations and discovers the role of neurons in the network. In addition to developing an understanding of our models, neuron analysis enables various applications such as debiasing, domain adaptation and architectural search. We present NeuroX, a comprehensive open-source toolkit to conduct neuron analysis of natural language processing models. It implements various interpretation methods under a unified API, and provides a framework for data processing and evaluation, thus making it easier for researchers and practitioners to perform neuron analysis. The Python toolkit is available at https://www.github.com/fdalvi/NeuroX. Demo Video available at https://youtu.be/mLhs2YMx4u8.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 32 of its 40 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

32 of 40 references.

References · 32

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  3. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  4. 2019Designing and Interpreting Probes with Control Taskscited by 640
  5. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
  6. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  7. 2019The What-If Tool: Interactive Probing of Machine Learning Modelscited by 207
  8. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  9. 2020Compositional Explanations of Neuronscited by 172
  10. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  11. 2015Visualizing and Understanding Recurrent Networkscited by 160
  12. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
20 more
  1. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  2. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 111
  3. 2020Explaining Explanations: Axiomatic Feature Interactions for Deep Networkscited by 103
  4. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  5. 2019The emergence of number and syntax units in LSTM language modelscited by 94
  6. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  7. 2018How Important Is a Neuron?cited by 73
  8. 2021On the Pitfalls of Analyzing Individual Neurons in Language Modelscited by 62
  9. 2020Intrinsic Probing through Dimension Selectioncited by 45
  10. 2016Representation of linguistic form and function in recurrent neural networkscited by 39
  11. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
  12. 2018NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networkscited by 31
  13. 2020Finding Experts in Transformer Modelscited by 28
  14. 2022LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Modelscited by 28
  15. 2022Discovering Salient Neurons in Deep NLP Modelscited by 21
  16. 2021Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translationcited by 15
  17. 2023Evaluating Neuron Interpretation Methods of NLP Modelscited by 8
  18. 2020diagNNose: A Library for Neural Activation Analysiscited by 7
  19. 2022ConceptX: A Framework for Latent Concept Analysiscited by 6
  20. 2023NxPlain: Web-based Tool for Discovery of Latent Conceptscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 3; 2026: 2.

11 papers.

  1. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  2. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  3. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  4. 2024Latent Concept-based Explanation of NLP Modelscited by 13
  5. 2024Exploring Alignment in Shared Cross-lingual Spacescited by 13
  6. 2023Scaling up Discovery of Latent Concepts in Deep NLP Modelscited by 9
  7. 2025Interpreting the Effects of Quantization on LLMscited by 5
  8. 2025Cross-Layer Discrete Concept Discovery for Interpreting Language Modelscited by 2
  9. 2025From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Modelscited by 1
  10. 2026Vector Quantized Latent Concepts: A Scalable Alternative to Clustering-Based Concept Discoverycited by 1
  11. 2026SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encodingcited by 0