# Neural Network Module Decomposition and Recomposition

arXiv: https://arxiv.org/abs/2112.13208 · HTML: https://mlsanity.com/paper/2112.13208/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 25 Dec 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [What's Hidden in a Randomly Weighted Neural Network?](https://mlsanity.com/paper/1911.13299/) (2019)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)
- [Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask](https://mlsanity.com/paper/1905.01067/) (2019)
- [Convolutional neural networks with low-rank regularization](https://mlsanity.com/paper/1511.06067/) (2015)
- [Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks](https://mlsanity.com/paper/2010.02066/) (2020)
- [A Review of Modularization Techniques in Artificial Neural Networks](https://mlsanity.com/paper/1904.12770/) (2019)
- [Modular Representation of Layered Neural Networks](https://mlsanity.com/paper/1703.00168/) (2017)
- [Investigating the Compositional Structure Of Deep Neural Networks](https://mlsanity.com/paper/2002.06967/) (2020)

## Built on by (most cited first, 10 of 10)

- [LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition](https://mlsanity.com/paper/2307.13269/) (2023)
- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Combining Modular Skills in Multitask Learning](https://mlsanity.com/paper/2202.13914/) (2022)
- [Customizable Combination of Parameter-Efficient Modules for Multi-Task Learning](https://mlsanity.com/paper/2312.03248/) (2023)
- [Patching Weak Convolutional Neural Network Models through Modularization and Composition](https://mlsanity.com/paper/2209.06116/) (2022)
- [Modularity in Deep Learning: A Survey](https://mlsanity.com/paper/2310.01154/) (2023)
- [Assessing the Portability of Parameter Matrices Trained by Parameter-Efficient Finetuning Methods](https://mlsanity.com/paper/2401.14228/) (2024)
- [Tensor-Aware Energy Accounting](https://mlsanity.com/paper/2311.11424/) (2023)
- [MedForge: Building Medical Foundation Models Like Open Source Software Development](https://mlsanity.com/paper/2502.16055/) (2025)
- [CircuitTree: Composing Circuits into A Syllable Transcriber](https://mlsanity.com/paper/2610.12520/) (2026)

## Abstract (arXiv)

We propose a modularization method that decomposes a deep neural network (DNN) into small modules from a functionality perspective and recomposes them into a new model for some other task. Decomposed modules are expected to have the advantages of interpretability and verifiability due to their small size. In contrast to existing studies based on reusing models that involve retraining, such as a transfer learning model, the proposed method does not require retraining and has wide applicability as it can be easily combined with existing functional modules. The proposed method extracts modules using weight masks and can be applied to arbitrary DNNs. Unlike existing studies, it requires no assumption about the network architecture. To extract modules, we designed a learning method and a loss function to maximize shared weights among modules. As a result, the extracted modules can be recomposed without a large increase in the size. We demonstrate that the proposed method can decompose and recompose DNNs with high compression ratio and high accuracy and is superior to the existing method through sharing weights between modules.