DNN Modularization via Activation-Driven Training
Not on the map. Why?
Abstract (arXiv)
Deep Neural Networks (DNNs) tend to accrue technical debt and suffer from significant retraining costs when adapting to evolving requirements. Modularizing DNNs offers the promise of improving their reusability. Previous work has proposed techniques to decompose DNN models into modules both during and after training. However, these strategies yield several shortcomings, including significant weight overlaps and accuracy losses across modules, restricted focus on convolutional layers only, and added complexity and training time by introducing auxiliary masks to control modularity. In this work, we propose MODA, an activation-driven modular training approach. MODA promotes inherent modularity within a DNN model by directly regulating the activation outputs of its layers based on three modular objectives: intra-class affinity, inter-class dispersion, and compactness. MODA is evaluated using three well-known DNN models and five datasets with varying sizes. This evaluation indicates that, compared to the existing state-of-the-art, using MODA yields several advantages: (1) MODA accomplishes modularization with 22% less training time; (2) the resultant modules generated by MODA comprise up to 24x fewer weights and 37x less weight overlap while (3) preserving the original model's accuracy without additional fine-tuning; in module replacement scenarios, (4) MODA improves the accuracy of a target class by 12% on average while ensuring minimal impact on the accuracy of other classes.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 33 of its 68 references that are in the map. First posted on arXiv 1 Nov 2024.
Builds on
33 of 68 references.
References · 33
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2016Pruning Filters for Efficient ConvNetscited by 1,143
- 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,143
- 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 744
- 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
- 2021A White Paper on Neural Network Quantizationcited by 472
- 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
21 more
- 2015Neural Module Networkscited by 376
- 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 311
- 2016Safety Verification of Deep Neural Networkscited by 263
- 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 251
- 2017Formal Verification of Piece-Wise Linear Feed-Forward Neural Networkscited by 242
- 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 163
- 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 147
- 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 98
- 2018AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inferencecited by 77
- 2016Towards Better Analysis of Deep Convolutional Neural Networkscited by 51
- 2019Arachne: Search Based Repair of Deep Neural Networkscited by 26
- 2020Discovering Parametric Activation Functionscited by 22
- 2019Impact of Fully Connected Layers on Performance of Convolutional Neural Networks for Image Classificationcited by 19
- 2021How Modular Should Neural Module Networks Be for Systematic Generalization?cited by 15
- 2021Decomposing Convolutional Neural Networks into Reusable and Replaceable Modulescited by 14
- 2019Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Goingcited by 13
- 2020Supporting DNN Safety Analysis and Retraining through Heatmap-based Unsupervised Learningcited by 11
- 2022Patching Weak Convolutional Neural Network Models through Modularization and Compositioncited by 6
- 2023Reusing Deep Neural Network Models through Model Re-engineeringcited by 6
- 2023Modularizing while Training: A New Paradigm for Modularizing DNN Modelscited by 4
- 2023Reusing Convolutional Neural Network Models through Modularization and Compositioncited by 3
Built on by
Citing papers per year: 2026: 3.
3 papers.