# Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability

arXiv: https://arxiv.org/abs/2305.08746 · HTML: https://mlsanity.com/paper/2305.08746/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability.
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 4 May 2023.

## Placed in

- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Discovering Symbolic Models from Deep Learning with Inductive Biases](https://mlsanity.com/paper/2006.11287/) (2020)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Structured Pruning of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1512.08571/) (2015)
- [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022)
- [AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity](https://mlsanity.com/paper/2006.10782/) (2020)
- [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks](https://mlsanity.com/paper/2010.02066/) (2020)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Modular Networks: Learning to Decompose Neural Computation](https://mlsanity.com/paper/1811.05249/) (2018)
- [Clusterability in Neural Networks](https://mlsanity.com/paper/2103.03386/) (2021)
- [The evolutionary origins of hierarchy](https://mlsanity.com/paper/1505.06353/) (2015)

## Built on by (most cited first, 30 of 38)

- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [KAN 2.0: Kolmogorov-Arnold Networks Meet Science](https://mlsanity.com/paper/2408.10205/) (2024)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [Codebook Features: Sparse and Discrete Interpretability for Neural Networks](https://mlsanity.com/paper/2310.17230/) (2023)
- [Mixture of Experts Made Intrinsically Interpretable](https://mlsanity.com/paper/2503.07639/) (2025)
- [Extreme sparsification of physics-augmented neural networks for interpretable model discovery in mechanics](https://mlsanity.com/paper/2310.03652/) (2023)
- [Using Degeneracy in the Loss Landscape for Mechanistic Interpretability](https://mlsanity.com/paper/2405.10927/) (2024)
- [Growing Brains: Co-emergence of Anatomical and Functional Modularity in Recurrent Neural Networks](https://mlsanity.com/paper/2310.07711/) (2023)
- [Low-Rank Adapting Models for Sparse Autoencoders](https://mlsanity.com/paper/2501.19406/) (2025)
- [Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks](https://mlsanity.com/paper/2511.19265/) (2025)
- [A Neural Scaling Law from Lottery Ticket Ensembling](https://mlsanity.com/paper/2310.02258/) (2023)
- [Machine Learning meets Algebraic Combinatorics: A Suite of Datasets Capturing Research-level Conjecturing Ability in Pure Mathematics](https://mlsanity.com/paper/2503.06366/) (2025)
- [The semantic landscape paradigm for neural networks](https://mlsanity.com/paper/2307.09550/) (2023)
- [Neural Network Pruning by Gradient Descent](https://mlsanity.com/paper/2311.12526/) (2023)
- [Efficient Model Compression Techniques with FishLeg](https://mlsanity.com/paper/2412.02328/) (2024)
- [Harmonic Loss Trains Interpretable AI Models](https://mlsanity.com/paper/2502.01628/) (2025)
- [Generating Interpretable Networks using Hypernetworks](https://mlsanity.com/paper/2312.03051/) (2023)
- [Quantum Algorithms for Compositional Text Processing](https://mlsanity.com/paper/2408.06061/) (2024)
- [The Dual-Stream Transformer: Channelized Architecture for Interpretable Language Modeling](https://mlsanity.com/paper/2603.07461/) (2026)
- [SPADE: Sparsity-Guided Debugging for Deep Neural Networks](https://mlsanity.com/paper/2310.04519/) (2023)
- [Hyperdimensional computing: a fast, robust and interpretable paradigm for biological data](https://mlsanity.com/paper/2402.17572/) (2024)
- [Machines and Mathematical Mutations: Using GNNs to Characterize Quiver Mutation Classes](https://mlsanity.com/paper/2411.07467/) (2024)
- [Modularity is the Bedrock of Natural and Artificial Intelligence](https://mlsanity.com/paper/2602.18960/) (2026)
- [The Garden of Forking Paths: Observing Dynamic Parameters Distribution in Large Language Models](https://mlsanity.com/paper/2403.08739/) (2024)
- [NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Models](https://mlsanity.com/paper/2404.01306/) (2024)
- [Training Neural Networks for Modularity aids Interpretability](https://mlsanity.com/paper/2409.15747/) (2024)
- [Inducing, Detecting and Characterising Neural Modules: A Pipeline for Functional Interpretability in Reinforcement Learning](https://mlsanity.com/paper/2501.17077/) (2025)

## Abstract (arXiv)

We introduce Brain-Inspired Modular Training (BIMT), a method for making neural networks more modular and interpretable. Inspired by brains, BIMT embeds neurons in a geometric space and augments the loss function with a cost proportional to the length of each neuron connection. We demonstrate that BIMT discovers useful modular neural networks for many simple tasks, revealing compositional structures in symbolic formulas, interpretable decision boundaries and features for classification, and mathematical structure in algorithmic datasets. The ability to directly see modules with the naked eye can complement current mechanistic interpretability strategies such as probes, interventions or staring at all weights.