# NeuroSurgeon: A Toolkit for Subnetwork Analysis

arXiv: https://arxiv.org/abs/2309.00244 · HTML: https://mlsanity.com/paper/2309.00244/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 25 references that are in the map.
- First posted on arXiv 1 Sep 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Parameter-Efficient Transfer Learning with Diff Pruning](https://mlsanity.com/paper/2012.07463/) (2020)
- [Task-Specific Skill Localization in Fine-tuned Language Models](https://mlsanity.com/paper/2302.06600/) (2023)
- [Winning the Lottery with Continuous Sparsification](https://mlsanity.com/paper/1912.04427/) (2019)
- [Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks](https://mlsanity.com/paper/2010.02066/) (2020)
- [Language Models Implement Simple Word2Vec-style Vector Arithmetic](https://mlsanity.com/paper/2305.16130/) (2023)
- [Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?](https://mlsanity.com/paper/2106.02890/) (2021)
- [Low-Complexity Probing via Finding Subnetworks](https://mlsanity.com/paper/2104.03514/) (2021)
- [How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking](https://mlsanity.com/paper/2004.14992/) (2020)
- [Break It Down: Evidence for Structural Compositionality in Neural Networks](https://mlsanity.com/paper/2301.10884/) (2023)
- [Sparse Interventions in Language Models with Differentiable Masking](https://mlsanity.com/paper/2112.06837/) (2021)
- [Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models](https://mlsanity.com/paper/2307.15049/) (2023)

## Built on by (most cited first, 9 of 9)

- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [Understanding Visual Feature Reliance through the Lens of Complexity](https://mlsanity.com/paper/2407.06076/) (2024)
- [Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects](https://mlsanity.com/paper/2406.15955/) (2024)
- [Uncovering Intermediate Variables in Transformers using Circuit Probing](https://mlsanity.com/paper/2311.04354/) (2023)
- [The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models](https://mlsanity.com/paper/2403.03942/) (2024)
- [Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models](https://mlsanity.com/paper/2504.15471/) (2025)
- [Bi-Orthogonal Factor Decomposition for Vision Transformers](https://mlsanity.com/paper/2601.05328/) (2026)
- [BrainSurgery: Reproducible and Reliable Declarative Weight Manipulations for Model Editing and Upcycling](https://mlsanity.com/paper/2606.09707/) (2026)

## Abstract (arXiv)

Despite recent advances in the field of explainability, much remains unknown about the algorithms that neural networks learn to represent. Recent work has attempted to understand trained models by decomposing them into functional circuits (Csord'as et al., 2020; Lepori et al., 2023). To advance this research, we developed NeuroSurgeon, a python library that can be used to discover and manipulate subnetworks within models in the Huggingface Transformers library (Wolf et al., 2019). NeuroSurgeon is freely available at https://github.com/mlepori1/NeuroSurgeon.