# Learning to Compose Neural Networks for Question Answering

arXiv: https://arxiv.org/abs/1601.01705 · HTML: https://mlsanity.com/paper/1601.01705/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2016.

## Built on by (most cited first, 30 of 85)

- [Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](https://mlsanity.com/paper/2304.09842/) (2023)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Hypothesis Search: Inductive Reasoning with Language Models](https://mlsanity.com/paper/2309.05660/) (2023)
- [NeurASP: Embracing Neural Networks into Answer Set Programming](https://mlsanity.com/paper/2307.07700/) (2023)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Continual Learning via Local Module Composition](https://mlsanity.com/paper/2111.07736/) (2021)
- [NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations](https://mlsanity.com/paper/2303.13483/) (2023)
- [Complex Knowledge Base Question Answering: A Survey](https://mlsanity.com/paper/2108.06688/) (2021)
- [Modular Visual Question Answering via Code Generation](https://mlsanity.com/paper/2306.05392/) (2023)
- [Neural Approaches to Conversational Information Retrieval](https://mlsanity.com/paper/2201.05176/) (2022)
- [On the Explainability of Natural Language Processing Deep Models](https://mlsanity.com/paper/2210.06929/) (2022)
- [Understanding Interlocking Dynamics of Cooperative Rationalization](https://mlsanity.com/paper/2110.13880/) (2021)
- [Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computing](https://mlsanity.com/paper/2210.15889/) (2022)
- [What's Left? Concept Grounding with Logic-Enhanced Foundation Models](https://mlsanity.com/paper/2310.16035/) (2023)
- [Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?](https://mlsanity.com/paper/2203.17271/) (2022)
- [On the Challenges and Opportunities in Generative AI](https://mlsanity.com/paper/2403.00025/) (2024)
- [Dissociating language and thought in large language models](https://mlsanity.com/paper/2301.06627/) (2023)
- [The Relational Bottleneck as an Inductive Bias for Efficient Abstraction](https://mlsanity.com/paper/2309.06629/) (2023)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiability](https://mlsanity.com/paper/2112.09054/) (2021)
- [GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs](https://mlsanity.com/paper/2311.04901/) (2023)
- [Distribution Matching for Rationalization](https://mlsanity.com/paper/2106.00320/) (2021)
- [Cognitive Visual Commonsense Reasoning Using Dynamic Working Memory](https://mlsanity.com/paper/2107.01671/) (2021)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [Neurosymbolic Grounding for Compositional World Models](https://mlsanity.com/paper/2310.12690/) (2023)
- [API-Assisted Code Generation for Question Answering on Varied Table Structures](https://mlsanity.com/paper/2310.14687/) (2023)
- [Examining Modularity in Multilingual LMs via Language-Specialized Subnetworks](https://mlsanity.com/paper/2311.08273/) (2023)

## Abstract (arXiv)

We describe a question answering model that applies to both images and structured knowledge bases. The model uses natural language strings to automatically assemble neural networks from a collection of composable modules. Parameters for these modules are learned jointly with network-assembly parameters via reinforcement learning, with only (world, question, answer) triples as supervision. Our approach, which we term a dynamic neural model network, achieves state-of-the-art results on benchmark datasets in both visual and structured domains.