mlsanityData to 7 Oct 2026

Routing Networks and the Challenges of Modular and Compositional Computation

arXiv 1904.12774 ↗First posted 29 Apr 2019cs.LG cs.NE stat.MLcited by 49

Not on the map. Why?

Abstract (arXiv)

Compositionality is a key strategy for addressing combinatorial complexity and the curse of dimensionality. Recent work has shown that compositional solutions can be learned and offer substantial gains across a variety of domains, including multi-task learning, language modeling, visual question answering, machine comprehension, and others. However, such models present unique challenges during training when both the module parameters and their composition must be learned jointly. In this paper, we identify several of these issues and analyze their underlying causes. Our discussion focuses on routing networks, a general approach to this problem, and examines empirically the interplay of these challenges and a variety of design decisions. In particular, we consider the effect of how the algorithm decides on module composition, how the algorithm updates the modules, and if the algorithm uses regularization.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 12; 2023: 7; 2024: 12; 2025: 7; 2026: 1.

49 papers.

Sort:

By area:

  1. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  2. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  3. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  4. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  5. 2021Coordination Among Neural Modules Through a Shared Global Workspacecited by 83
  6. 2023Modular Deep Learningcited by 82
  7. 2021Neural Production Systems: Learning Rule-Governed Visual Dynamicscited by 75
  8. 2021Continual Learning via Local Module Compositioncited by 61
  9. 2021Discrete-Valued Neural Communicationcited by 39
  10. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  11. 2021Fast and Slow Learning of Recurrent Independent Mechanismscited by 36
  12. 2022Is a Modular Architecture Enough?cited by 35
37 more
  1. 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
  2. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  3. 2024Position: Theory of Mind Benchmarks are Broken for Large Language Modelscited by 30
  4. 2022Combining Modular Skills in Multitask Learningcited by 27
  5. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  6. 2021Dynamic Inference with Neural Interpreterscited by 24
  7. 2022Spot-adaptive Knowledge Distillationcited by 22
  8. 2024Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulationcited by 15
  9. 2022Continual Learning for Multivariate Time Series Tasks with Variable Input Dimensionscited by 14
  10. 2025Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Modelscited by 13
  11. 2024Learning More Generalized Experts by Merging Experts in Mixture-of-Expertscited by 12
  12. 2021Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Expertscited by 10
  13. 2023Revisiting Single-gated Mixtures of Expertscited by 10
  14. 2025Self-Composing Policies for Scalable Continual Reinforcement Learningcited by 10
  15. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  16. 2025On-device edge learning for IoT data streams: a surveycited by 8
  17. 2024Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inferencecited by 7
  18. 2023QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharingcited by 6
  19. 2025Bridging Streaming Continual Learning via In-Context Large Tabular Modelscited by 6
  20. 2024What makes Models Compositional? A Theoretical View: With Supplementcited by 5
  21. 2024Breaking Neural Network Scaling Laws with Modularitycited by 5
  22. 2024LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashingcited by 5
  23. 2021Efficient and robust multi-task learning in the brain with modular latent primitivescited by 4
  24. 2023Reusable Slotwise Mechanismscited by 4
  25. 2023Compositional Program Generation for Few-Shot Systematic Generalizationcited by 4
  26. 2022Learning Label Modular Prompts for Text Classification in the Wildcited by 3
  27. 2023Modularity in Deep Learning: A Surveycited by 3
  28. 2025Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environmentscited by 3
  29. 2024More Experts Than Galaxies: Conditionally-overlapping Experts With Biologically-Inspired Fixed Routingcited by 2
  30. 2025Compositional meta-learning through probabilistic task inferencecited by 2
  31. 2025The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Modelscited by 2
  32. 2024Remembering Transformer for Continual Learningcited by 1
  33. 2026Position: Collusion Risks Among AI Reasoning Agents Justify Certification Requirements for Making Market Decisionscited by 1
  34. 2021All-In-One: Artificial Association Neural Networkscited by 0
  35. 2022Towards Understanding the Link Between Modularity and Performance in Neural Networks for Reinforcement Learningcited by 0
  36. 2022A Short Survey of Systematic Generalizationcited by 0
  37. 2024CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Communitycited by 0