Modularity in Deep Learning: A Survey
Not on the map. Why?
Abstract (arXiv)
Modularity is a general principle present in many fields. It offers attractive advantages, including, among others, ease of conceptualization, interpretability, scalability, module combinability, and module reusability. The deep learning community has long sought to take inspiration from the modularity principle, either implicitly or explicitly. This interest has been increasing over recent years. We review the notion of modularity in deep learning around three axes: data, task, and model, which characterize the life cycle of deep learning. Data modularity refers to the observation or creation of data groups for various purposes. Task modularity refers to the decomposition of tasks into sub-tasks. Model modularity means that the architecture of a neural network system can be decomposed into identifiable modules. We describe different instantiations of the modularity principle, and we contextualize their advantages in different deep learning sub-fields. Finally, we conclude the paper with a discussion of the definition of modularity and directions for future research.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 147 of its 270 references that are in the map. First posted on arXiv 2 Oct 2023.
Builds on
147 of 270 references.
References · 147
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2022Generative Adversarial Networkscited by 5,603
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
38 more
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2016Wide Residual Networkscited by 3,057
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2017Prototypical Networks for Few-shot Learningcited by 3,001
- 2020Exploring Simple Siamese Representation Learningcited by 2,853
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
- 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
- 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
- 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2016Progressive Neural Networkscited by 1,332
- 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2014How transferable are features in deep neural networks?cited by 1,216
- 2020Gradient Surgery for Multi-Task Learningcited by 1,210
- 2016Group Equivariant Convolutional Networkscited by 1,154
- 2017Continual Learning with Deep Generative Replaycited by 1,152
- 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
- 2016An overview of gradient descent optimization algorithmscited by 1,021
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
- 2016HyperNetworkscited by 1,003
- 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
- 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
- 2017A Survey on Multi-Task Learningcited by 838
- 2015Deep Learning and the Information Bottleneck Principlecited by 793
- 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
- 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
- 2020A Survey of Deep Active Learningcited by 547
- 2021Fast Model Editing at Scalecited by 491
- 2022Memory-Based Model Editing at Scalecited by 443
- 2019General E(2)-Equivariant Steerable CNNscited by 441
- 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
50 most cited of 147 All 147 (JSON)
Built on by
Citing papers per year: 2024: 2; 2025: 0; 2026: 1.
3 papers.