# How Modular Should Neural Module Networks Be for Systematic Generalization?

arXiv: https://arxiv.org/abs/2106.08170 · HTML: https://mlsanity.com/paper/2106.08170/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [Inferring and Executing Programs for Visual Reasoning](https://mlsanity.com/paper/1705.03633/) (2017)
- [Learning to Reason: End-to-End Module Networks for Visual Question Answering](https://mlsanity.com/paper/1704.05526/) (2017)
- [Task-Driven Modular Networks for Zero-Shot Compositional Learning](https://mlsanity.com/paper/1905.05908/) (2019)
- [A Benchmark for Systematic Generalization in Grounded Language Understanding](https://mlsanity.com/paper/2003.05161/) (2020)
- [On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law](https://mlsanity.com/paper/2005.09241/) (2020)
- [CLOSURE: Assessing Systematic Generalization of CLEVR Models](https://mlsanity.com/paper/1912.05783/) (2019)
- [Visual Question Answering: Datasets, Algorithms, and Future Challenges](https://mlsanity.com/paper/1610.01465/) (2016)
- [Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks](https://mlsanity.com/paper/1807.07545/) (2018)
- [When and how CNNs generalize to out-of-distribution category-viewpoint combinations](https://mlsanity.com/paper/2007.08032/) (2020)
- [Neural Compositional Denotational Semantics for Question Answering](https://mlsanity.com/paper/1808.09942/) (2018)
- [Robustness to Transformations Across Categories: Is Robustness To Transformations Driven by Invariant Neural Representations?](https://mlsanity.com/paper/2007.00112/) (2020)

## Built on by (most cited first, 15 of 15)

- [How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition](https://mlsanity.com/paper/2207.07730/) (2022)
- [A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practices](https://mlsanity.com/paper/2406.08787/) (2024)
- [Improving Systematic Generalization Through Modularity and Augmentation](https://mlsanity.com/paper/2202.10745/) (2022)
- [Learning to Compose: Improving Object Centric Learning by Injecting Compositionality](https://mlsanity.com/paper/2405.00646/) (2024)
- [Object-Category Aware Reinforcement Learning](https://mlsanity.com/paper/2210.07802/) (2022)
- [Transformer Module Networks for Systematic Generalization in Visual Question Answering](https://mlsanity.com/paper/2201.11316/) (2022)
- [Breaking Neural Network Scaling Laws with Modularity](https://mlsanity.com/paper/2409.05780/) (2024)
- [Modularity in Deep Learning: A Survey](https://mlsanity.com/paper/2310.01154/) (2023)
- [DNN Modularization via Activation-Driven Training](https://mlsanity.com/paper/2411.01074/) (2024)
- [Improved Object-Centric Diffusion Learning with Registers and Contrastive Alignment](https://mlsanity.com/paper/2601.01224/) (2026)
- [D3: Data Diversity Design for Systematic Generalization in Visual Question Answering](https://mlsanity.com/paper/2309.08798/) (2023)
- [HICO-DET-SG and V-COCO-SG: New Data Splits for Evaluating the Systematic Generalization Performance of Human-Object Interaction Detection Models](https://mlsanity.com/paper/2305.09948/) (2023)
- [What You Can't See Is What You Learn: Slot-Selective Evidence Masking Favors Compositional Generalization in Shared-Genome Language-Model Societies](https://mlsanity.com/paper/2608.20054/) (2026)
- [Modularity Trumps Invariance for Compositional Robustness](https://mlsanity.com/paper/2306.09005/) (2023)
- [Disentanglement-Based Equivariant Learning for Compositional VQA](https://mlsanity.com/paper/2606.02168/) (2026)

## Abstract (arXiv)

Neural Module Networks (NMNs) aim at Visual Question Answering (VQA) via composition of modules that tackle a sub-task. NMNs are a promising strategy to achieve systematic generalization, i.e., overcoming biasing factors in the training distribution. However, the aspects of NMNs that facilitate systematic generalization are not fully understood. In this paper, we demonstrate that the degree of modularity of the NMN have large influence on systematic generalization. In a series of experiments on three VQA datasets (VQA-MNIST, SQOOP, and CLEVR-CoGenT), our results reveal that tuning the degree of modularity, especially at the image encoder stage, reaches substantially higher systematic generalization. These findings lead to new NMN architectures that outperform previous ones in terms of systematic generalization.