# Studying Cross-cluster Modularity in Neural Networks

arXiv: https://arxiv.org/abs/2502.02470 · HTML: https://mlsanity.com/paper/2502.02470/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Builds on (most cited first, 23 of 23)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)
- [MoEfication: Transformer Feed-forward Layers are Mixtures of Experts](https://mlsanity.com/paper/2110.01786/) (2021)
- [Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla](https://mlsanity.com/paper/2307.09458/) (2023)
- [Efficient Continual Learning with Modular Networks and Task-Driven Priors](https://mlsanity.com/paper/2012.12631/) (2020)
- [Patent Claim Generation by Fine-Tuning OpenAI GPT-2](https://mlsanity.com/paper/1907.02052/) (2019)
- [Gradient Routing: Masking Gradients to Localize Computation in Neural Networks](https://mlsanity.com/paper/2410.04332/) (2024)
- [Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification](https://mlsanity.com/paper/2011.04041/) (2020)
- [Clusterability in Neural Networks](https://mlsanity.com/paper/2103.03386/) (2021)
- [Modularity-Aware Graph Autoencoders for Joint Community Detection and Link Prediction](https://mlsanity.com/paper/2202.00961/) (2022)
- [Monet: Mixture of Monosemantic Experts for Transformers](https://mlsanity.com/paper/2412.04139/) (2024)
- [Modular Representation of Layered Neural Networks](https://mlsanity.com/paper/1703.00168/) (2017)
- [Clustering units in neural networks: upstream vs downstream information](https://mlsanity.com/paper/2203.11815/) (2022)
- [Improving the Interpretability of fMRI Decoding using Deep Neural Networks and Adversarial Robustness](https://mlsanity.com/paper/2004.11114/) (2020)
- [Deriving pairwise transfer entropy from network structure and motifs](https://mlsanity.com/paper/1911.02931/) (2019)

## Built on by (most cited first, 2 of 2)

- [Modular Pretraining Enables Access Control](https://mlsanity.com/paper/2607.08077/) (2026)
- [Sparse Competition during Training For the Emergence of Specialized Modules](https://mlsanity.com/paper/2608.30978/) (2026)

## Abstract (arXiv)

An approach to improve neural network interpretability is via clusterability, i.e., splitting a model into disjoint clusters that can be studied independently. We define a measure for clusterability and show that pre-trained models form highly enmeshed clusters via spectral graph clustering. We thus train models to be more modular using a "clusterability loss" function that encourages the formation of non-interacting clusters. We then investigate the emerging properties of these highly clustered models. We find our trained clustered models do not exhibit more task specialization, but do form smaller circuits. We investigate CNNs trained on MNIST and CIFAR, small transformers trained on modular addition, and GPT-2 and Pythia on the Wiki dataset, and Gemma on a Chemistry dataset. This investigation shows what to expect from clustered models.