mlsanityData to 7 Oct 2026

Modular Deep Learning

arXiv 2302.11529 ↗First posted 22 Feb 2023cs.LGcited by 82

Not on the map. Why?

Abstract (arXiv)

Transfer learning has recently become the dominant paradigm of machine learning. Pre-trained models fine-tuned for downstream tasks achieve better performance with fewer labelled examples. Nonetheless, it remains unclear how to develop models that specialise towards multiple tasks without incurring negative interference and that generalise systematically to non-identically distributed tasks. Modular deep learning has emerged as a promising solution to these challenges. In this framework, units of computation are often implemented as autonomous parameter-efficient modules. Information is conditionally routed to a subset of modules and subsequently aggregated. These properties enable positive transfer and systematic generalisation by separating computation from routing and updating modules locally. We offer a survey of modular architectures, providing a unified view over several threads of research that evolved independently in the scientific literature. Moreover, we explore various additional purposes of modularity, including scaling language models, causal inference, programme induction, and planning in reinforcement learning. Finally, we report various concrete applications where modularity has been successfully deployed such as cross-lingual and cross-modal knowledge transfer. Related talks and projects to this survey, are available at https://www.modulardeeplearning.com/.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 256 of its 329 references that are in the map. First posted on arXiv 22 Feb 2023.

Builds on

256 of 329 references.

References · 256

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  11. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  12. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  13. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  14. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  15. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  16. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  17. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  18. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  19. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  20. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  21. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  22. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  23. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  24. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  25. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  26. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  27. 2022Editing Models with Task Arithmeticcited by 1,076
  28. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  29. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  30. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  31. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  32. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  33. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  34. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  35. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  36. 2020Object-Centric Learning with Slot Attentioncited by 865
  37. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  38. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844

50 most cited of 256 All 256 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 27; 2025: 22; 2026: 10.

82 papers.

Sort:

By area:

  1. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  2. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  3. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  4. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  5. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  6. 2023Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretabilitycited by 38
  7. 2023Elastic Weight Removal for Faithful and Abstractive Dialogue Generationcited by 29
  8. 2023Denoising Task Routing for Diffusion Modelscited by 28
  9. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  10. 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Expertscited by 25
  11. 2024Modular Blind Video Quality Assessmentcited by 23
  12. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
38 more
  1. 2023Romanization-based Large-scale Adaptation of Multilingual Language Modelscited by 21
  2. 2024Fine-tuning Large Language Models with Sequential Instructionscited by 20
  3. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18
  4. 2024A Second-Order Perspective on Model Compositionality and Incremental Learningcited by 18
  5. 2025A Survey of Continual Reinforcement Learningcited by 16
  6. 2024Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selectioncited by 13
  7. 2024A Complexity-Based Theory of Compositionalitycited by 13
  8. 2024Conditional computation in neural networks: principles and research trendscited by 10
  9. 2025Mixture of Cognitive Reasoners: Modular Reasoning with Brain-Like Specializationcited by 9
  10. 2025Physics of Skill Learningcited by 8
  11. 2025Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMscited by 8
  12. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  13. 2023Modular Domain Adaptation for Conformer-Based Streaming ASRcited by 7
  14. 2023Adaptivity and Modularity for Efficient Generalization Over Task Complexitycited by 7
  15. 2023MixtureGrowth: Growing Neural Networks by Recombining Learned Parameterscited by 7
  16. 2024The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insightscited by 7
  17. 2024ELICIT: LLM Augmentation via External In-Context Capabilitycited by 7
  18. 2023Distilling Efficient Language-Specific Models for Cross-Lingual Transfercited by 6
  19. 2023Adaptive Computation Modules: Granular Conditional Computation For Efficient Inferencecited by 6
  20. 2024Pheme: Efficient and Conversational Speech Generationcited by 6
  21. 2023Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settingscited by 5
  22. 2024Investigating Recurrent Transformers with Dynamic Haltcited by 5
  23. 2024m2mKD: Module-to-Module Knowledge Distillation for Modular Transformerscited by 5
  24. 2024No Train but Gain: Language Arithmetic for training-free Language Adapters enhancementcited by 5
  25. 2024Breaking Neural Network Scaling Laws with Modularitycited by 5
  26. 2024Is Multiple Object Tracking a Matter of Specialization?cited by 5
  27. 2023PEFT-Ref: A Modular Reference Architecture and Typology for Parameter-Efficient Finetuning Techniquescited by 4
  28. 2023Transfer-Free Data-Efficient Multilingual Slot Labelingcited by 4
  29. 2024Investigating Neuron Ablation in Attention Heads: The Case for Peak Activation Centeringcited by 4
  30. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  31. 2025Modular Machine Learning: An Indispensable Path towards New-Generation Large Language Modelscited by 4
  32. 2025Realizing Scaling Laws in Recommender Systems: A Foundation-Expert Paradigm for Hyperscale Model Deploymentcited by 4
  33. 2025Structural Priors and Modular Adapters in the Composable Fine-Tuning Algorithm of Large-Scale Modelscited by 4
  34. 2026A Theoretical Framework for Modular Learning of Robust Generative Modelscited by 4
  35. 2025DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detectioncited by 3
  36. 2025Context Parametrization with Compositional Adapterscited by 3
  37. 2023ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scalecited by 2
  38. 2024SuperPos-Prompt: Enhancing Soft Prompt Tuning of Language Models with Superposition of Multi Token Embeddingscited by 2

50 most cited of 82 All 82 (JSON)