# Clusterability in Neural Networks

arXiv: https://arxiv.org/abs/2103.03386 · HTML: https://mlsanity.com/paper/2103.03386/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2021.

## Built on by (most cited first, 24 of 24)

- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Visual Representation Learning Does Not Generalize Strongly Within the Same Domain](https://mlsanity.com/paper/2107.08221/) (2021)
- [Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient](https://mlsanity.com/paper/2410.02984/) (2024)
- [Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability](https://mlsanity.com/paper/2305.08746/) (2023)
- [Clustering units in neural networks: upstream vs downstream information](https://mlsanity.com/paper/2203.11815/) (2022)
- [Quantifying Local Specialization in Deep Neural Networks](https://mlsanity.com/paper/2110.08058/) (2021)
- [Graph-Based Similarity of Neural Network Representations](https://mlsanity.com/paper/2111.11165/) (2021)
- [Dynamics of specialization in neural modules under resource constraints](https://mlsanity.com/paper/2106.02626/) (2021)
- [Modularity in Reinforcement Learning via Algorithmic Independence in Credit Assignment](https://mlsanity.com/paper/2106.14993/) (2021)
- [Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behavior](https://mlsanity.com/paper/2206.13498/) (2022)
- [Pruning for Feature-Preserving Circuits in CNNs](https://mlsanity.com/paper/2206.01627/) (2022)
- [Modular Federated Learning](https://mlsanity.com/paper/2209.03090/) (2022)
- [Modularity in Deep Learning: A Survey](https://mlsanity.com/paper/2310.01154/) (2023)
- [Understanding Sparse Neural Networks from their Topology via Multipartite Graph Representations](https://mlsanity.com/paper/2305.16886/) (2023)
- [Studying Cross-cluster Modularity in Neural Networks](https://mlsanity.com/paper/2502.02470/) (2025)
- [Evaluating Explanations: An Explanatory Virtues Framework for Mechanistic Interpretability -- The Strange Science Part I.ii](https://mlsanity.com/paper/2505.01372/) (2025)
- [Leveraging the Graph Structure of Neural Network Training Dynamics](https://mlsanity.com/paper/2111.05410/) (2021)
- [Geometric sparsification in recurrent neural networks](https://mlsanity.com/paper/2406.06290/) (2024)
- [Training Neural Networks for Modularity aids Interpretability](https://mlsanity.com/paper/2409.15747/) (2024)
- [Inducing, Detecting and Characterising Neural Modules: A Pipeline for Functional Interpretability in Reinforcement Learning](https://mlsanity.com/paper/2501.17077/) (2025)
- [AI Model Utilization Measurements For Finding Class Encoding Patterns](https://mlsanity.com/paper/2212.06576/) (2022)
- [Neural Sculpting: Uncovering hierarchically modular task structure in neural networks through pruning and network analysis](https://mlsanity.com/paper/2305.18402/) (2023)
- [Sparse Competition during Training For the Emergence of Specialized Modules](https://mlsanity.com/paper/2608.30978/) (2026)
- [RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing](https://mlsanity.com/paper/2610.11775/) (2026)

## Abstract (arXiv)

The learned weights of a neural network have often been considered devoid of scrutable internal structure. In this paper, however, we look for structure in the form of clusterability: how well a network can be divided into groups of neurons with strong internal connectivity but weak external connectivity. We find that a trained neural network is typically more clusterable than randomly initialized networks, and often clusterable relative to random networks with the same distribution of weights. We also exhibit novel methods to promote clusterability in neural network training, and find that in multi-layer perceptrons they lead to more clusterable networks with little reduction in accuracy. Understanding and controlling the clusterability of neural networks will hopefully render their inner workings more interpretable to engineers by facilitating partitioning into meaningful clusters.