mlsanityData to 7 Oct 2026

Learning to Reason: End-to-End Module Networks for Visual Question Answering

arXiv 1704.05526 ↗First posted 18 Apr 2017cs.CVcited by 125

Not on the map. Why?

Abstract (arXiv)

Natural language questions are inherently compositional, and many are most easily answered by reasoning about their decomposition into modular sub-problems. For example, to answer "is there an equal number of balls and boxes?" we can look for balls, look for boxes, count them, and compare the results. The recently proposed Neural Module Network (NMN) architecture implements this approach to question answering by parsing questions into linguistic substructures and assembling question-specific deep networks from smaller modules that each solve one subtask. However, existing NMN implementations rely on brittle off-the-shelf parsers, and are restricted to the module configurations proposed by these parsers rather than learning them from data. In this paper, we propose End-to-End Module Networks (N2NMNs), which learn to reason by directly predicting instance-specific network layouts without the aid of a parser. Our model learns to generate network structures (by imitating expert demonstrations) while simultaneously learning network parameters (using the downstream task loss). Experimental results on the new CLEVR dataset targeted at compositional question answering show that N2NMNs achieve an error reduction of nearly 50% relative to state-of-the-art attentional approaches, while discovering interpretable network architectures specialized for each question.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 39; 2022: 33; 2023: 29; 2024: 10; 2025: 12; 2026: 2.

125 papers.

Sort:

By area:

  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  3. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  4. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  5. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  6. 2022Binding Language Models in Symbolic Languagescited by 224
  7. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  8. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  9. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  10. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  11. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  12. 2023Modular Deep Learningcited by 82
38 more
  1. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  2. 2021Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answeringcited by 71
  3. 2021Neuro-Symbolic Artificial Intelligence: Current Trendscited by 54
  4. 2023Modular Visual Question Answering via Code Generationcited by 47
  5. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  6. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 46
  7. 2022On the Explainability of Natural Language Processing Deep Modelscited by 37
  8. 2021DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answeringcited by 35
  9. 2022Is a Modular Architecture Enough?cited by 35
  10. 2021Greedy Gradient Ensemble for Robust Visual Question Answeringcited by 34
  11. 2022Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computingcited by 34
  12. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  13. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  14. 2024TANGO: Training-free Embodied AI Agents for Open-world Taskscited by 26
  15. 2021Learning by Planning: Language-Guided Global Image Editingcited by 25
  16. 2021Separating Skills and Concepts for Novel Visual Question Answeringcited by 24
  17. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  18. 2021Iterated learning for emergent systematicity in VQAcited by 21
  19. 2023Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answeringcited by 21
  20. 2021Towers of Babel: Combining Images, Language, and 3D Geometry for Learning Multimodal Visioncited by 18
  21. 2022Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problemcited by 18
  22. 2021Towards Language-guided Visual Recognition via Dynamic Convolutionscited by 17
  23. 2022METGEN: A Module-Based Entailment Tree Generation Framework for Answer Explanationcited by 17
  24. 2021Who's Waldo? Linking People Across Text and Imagescited by 16
  25. 2021Auto-Parsing Network for Image Captioning and Visual Question Answeringcited by 16
  26. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  27. 2021How Modular Should Neural Module Networks Be for Systematic Generalization?cited by 15
  28. 2021Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?cited by 15
  29. 2022Measuring Compositional Consistency for Video Question Answeringcited by 15
  30. 2021Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Imagescited by 14
  31. 2021Decomposing Convolutional Neural Networks into Reusable and Replaceable Modulescited by 14
  32. 2022VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challengescited by 14
  33. 2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
  34. 2024On The Specialization of Neural Modulescited by 14
  35. 2021Graphhopper: Multi-Hop Scene Graph Reasoning for Visual Question Answeringcited by 13
  36. 2022A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answeringcited by 13
  37. 2023Look, Remember and Reason: Grounded reasoning in videos with language modelscited by 13
  38. 2025Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Enginescited by 13

50 most cited of 125 All 125 (JSON)