mlsanityData to 7 Oct 2026

Modular Networks: Learning to Decompose Neural Computation

arXiv 1811.05249 ↗First posted 13 Nov 2018cs.LG cs.AI stat.MLcited by 62

Not on the map. Why?

Abstract (arXiv)

Scaling model capacity has been vital in the success of deep learning. For a typical network, necessary compute resources and training time grow dramatically with model size. Conditional computation is a promising way to increase the number of parameters with a relatively small increase in resources. We propose a training algorithm that flexibly chooses neural modules based on the data to be processed. Both the decomposition and modules are learned end-to-end. In contrast to existing approaches, training does not rely on regularization to enforce diversity in module use. We apply modular networks both to image recognition and language modeling tasks, where we achieve superior performance compared to several baselines. Introspection reveals that modules specialize in interpretable contexts.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 17; 2023: 15; 2024: 7; 2025: 8; 2026: 3.

62 papers.

Sort:

By area:

  1. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  2. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  3. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  4. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  5. 2023Modular Deep Learningcited by 82
  6. 2021Neural Production Systems: Learning Rule-Governed Visual Dynamicscited by 75
  7. 2021Continual Learning via Local Module Compositioncited by 61
  8. 2022Modular Lifelong Reinforcement Learning via Neural Compositioncited by 39
  9. 2023Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretabilitycited by 38
  10. 2022Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matricescited by 37
  11. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  12. 2021Fast and Slow Learning of Recurrent Independent Mechanismscited by 36
38 more
  1. 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
  2. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  3. 2022Combining Modular Skills in Multitask Learningcited by 27
  4. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  5. 2021Understanding the Dynamics of DNNs Using Graph Modularitycited by 26
  6. 2021Dynamic Inference with Neural Interpreterscited by 24
  7. 2022Spot-adaptive Knowledge Distillationcited by 22
  8. 2022CompoSuite: A Compositional Reinforcement Learning Benchmarkcited by 21
  9. 2023Improving Expert Specialization in Mixture of Expertscited by 16
  10. 2023Scalable Geometric Fracture Assembly via Co-creation Space among Assemblerscited by 13
  11. 2024Semi-Supervised Semantic Segmentation Based on Pseudo-Labels: A Surveycited by 12
  12. 2024Attention as a Hypernetworkcited by 12
  13. 2021Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Expertscited by 10
  14. 2021Quantifying Local Specialization in Deep Neural Networkscited by 10
  15. 2023Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learningcited by 9
  16. 2025HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performancecited by 9
  17. 2021Compositional Learning-based Planning for Vision POMDPscited by 8
  18. 2022Personalized Federated Learning via Heterogeneous Modular Networkscited by 8
  19. 2025On-device edge learning for IoT data streams: a surveycited by 8
  20. 2025Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMscited by 8
  21. 2021Dynamics of specialization in neural modules under resource constraintscited by 6
  22. 2021Modularity in Reinforcement Learning via Algorithmic Independence in Credit Assignmentcited by 6
  23. 2022Patching Weak Convolutional Neural Network Models through Modularization and Compositioncited by 5
  24. 2022Pruning for Feature-Preserving Circuits in CNNscited by 4
  25. 2024Gradient-based inference of abstract task representations for generalization in neural networkscited by 4
  26. 2022Logical Tasks for Measuring Extrapolation and Rule Comprehensioncited by 3
  27. 2022Learning Label Modular Prompts for Text Classification in the Wildcited by 3
  28. 2023Modularity in Deep Learning: A Surveycited by 3
  29. 2025Forward Once for All: Structural Parameterized Adaptation for Efficient Cloud-coordinated On-device Recommendationcited by 3
  30. 2021Compositional Models: Multi-Task Learning and Knowledge Transfer with Modular Networkscited by 2
  31. 2022Modular Adaptive Policy Selection for Multi-Task Imitation Learning through Task Divisioncited by 2
  32. 2022Analysis of Branch Specialization and its Application in Image Decompositioncited by 2
  33. 2023On Conditional and Compositional Language Model Differentiable Promptingcited by 2
  34. 2024Flexible task abstractions emerge in linear networks with fast and bounded unitscited by 2
  35. 2026Modularity is the Bedrock of Natural and Artificial Intelligencecited by 2
  36. 2021A modularity comparison of Long Short-Term Memory and Morphognosis neural networkscited by 1
  37. 2023Computing with Categories in Machine Learningcited by 1
  38. 2023ECLM: Efficient Edge-Cloud Collaborative Learning with Continuous Environment Adaptationcited by 1

50 most cited of 62 All 62 (JSON)