# DNN Modularization via Activation-Driven Training

arXiv: https://arxiv.org/abs/2411.01074 · HTML: https://mlsanity.com/paper/2411.01074/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 68 references that are in the map.
- First posted on arXiv 1 Nov 2024.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks](https://mlsanity.com/paper/1702.01135/) (2017)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [Safety Verification of Deep Neural Networks](https://mlsanity.com/paper/1610.06940/) (2016)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Formal Verification of Piece-Wise Linear Feed-Forward Neural Networks](https://mlsanity.com/paper/1705.01320/) (2017)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks](https://mlsanity.com/paper/2010.02066/) (2020)
- [AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference](https://mlsanity.com/paper/1805.08941/) (2018)
- [Towards Better Analysis of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1604.07043/) (2016)
- [Arachne: Search Based Repair of Deep Neural Networks](https://mlsanity.com/paper/1912.12463/) (2019)
- [Discovering Parametric Activation Functions](https://mlsanity.com/paper/2006.03179/) (2020)
- [Impact of Fully Connected Layers on Performance of Convolutional Neural Networks for Image Classification](https://mlsanity.com/paper/1902.02771/) (2019)
- [How Modular Should Neural Module Networks Be for Systematic Generalization?](https://mlsanity.com/paper/2106.08170/) (2021)
- [Decomposing Convolutional Neural Networks into Reusable and Replaceable Modules](https://mlsanity.com/paper/2110.07720/) (2021)
- [Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going](https://mlsanity.com/paper/1901.06955/) (2019)
- [Supporting DNN Safety Analysis and Retraining through Heatmap-based Unsupervised Learning](https://mlsanity.com/paper/2002.00863/) (2020)
- [Patching Weak Convolutional Neural Network Models through Modularization and Composition](https://mlsanity.com/paper/2209.06116/) (2022)

## Built on by (most cited first, 3 of 3)

- [Catalyst: Out-of-Distribution Detection via Elastic Scaling](https://mlsanity.com/paper/2602.02409/) (2026)
- [CORTEX: Learning to Share and Specialize in Dense Language Models](https://mlsanity.com/paper/2609.34449/) (2026)
- [Mind the Gaps: From Failure Attribution to Closed-Form Repair of Code Language Models](https://mlsanity.com/paper/2610.05277/) (2026)

## Abstract (arXiv)

Deep Neural Networks (DNNs) tend to accrue technical debt and suffer from significant retraining costs when adapting to evolving requirements. Modularizing DNNs offers the promise of improving their reusability. Previous work has proposed techniques to decompose DNN models into modules both during and after training. However, these strategies yield several shortcomings, including significant weight overlaps and accuracy losses across modules, restricted focus on convolutional layers only, and added complexity and training time by introducing auxiliary masks to control modularity. In this work, we propose MODA, an activation-driven modular training approach. MODA promotes inherent modularity within a DNN model by directly regulating the activation outputs of its layers based on three modular objectives: intra-class affinity, inter-class dispersion, and compactness. MODA is evaluated using three well-known DNN models and five datasets with varying sizes. This evaluation indicates that, compared to the existing state-of-the-art, using MODA yields several advantages: (1) MODA accomplishes modularization with 22% less training time; (2) the resultant modules generated by MODA comprise up to 24x fewer weights and 37x less weight overlap while (3) preserving the original model's accuracy without additional fine-tuning; in module replacement scenarios, (4) MODA improves the accuracy of a target class by 12% on average while ensuring minimal impact on the accuracy of other classes.