mlsanityData to 14 Oct 2026

Inferring and Executing Programs for Visual Reasoning

arXiv 1705.03633 ↗First posted 10 May 2017cs.CV cs.CL cs.LGcited by 130

Not on the map. Why?

Cited by 28 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing methods for visual reasoning attempt to directly map inputs to outputs using black-box architectures without explicitly modeling the underlying reasoning processes. As a result, these black-box models often learn to exploit biases in the data rather than learning to perform visual reasoning. Inspired by module networks, this paper proposes a model for visual reasoning that consists of a program generator that constructs an explicit representation of the reasoning process to be performed, and an execution engine that executes the resulting program to produce an answer. Both the program generator and the execution engine are implemented by neural networks, and are trained using a combination of backpropagation and REINFORCE. Using the CLEVR benchmark for visual reasoning, we show that our model significantly outperforms strong baselines and generalizes better in a variety of settings.

Key facts. Cited by 130 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 39; 2022: 26; 2023: 28; 2024: 15; 2025: 15; 2026: 7.

130 papers.

Sort:

By area:

  1. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  2. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  3. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  4. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  5. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  6. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  7. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  8. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164
  9. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  10. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  11. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  12. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 96
38 more
  1. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 96
  2. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 79
  3. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  4. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  5. 2022Theory of Graph Neural Networks: Representation and Learningcited by 61
  6. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
  7. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  8. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  9. 2021Leveraging Language to Learn Program Abstractions and Search Heuristicscited by 45
  10. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Modelscited by 41
  11. 2021Fast and Slow Learning of Recurrent Independent Mechanismscited by 36
  12. 2021Understanding Interlocking Dynamics of Cooperative Rationalizationcited by 36
  13. 2022Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computingcited by 35
  14. 2023What's Left? Concept Grounding with Logic-Enhanced Foundation Modelscited by 33
  15. 2021Linguistic Structures as Weak Supervision for Visual Scene Graph Generationcited by 29
  16. 2023Dissociating language and thought in large language modelscited by 29
  17. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  18. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 28
  19. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  20. 2023The Relational Bottleneck as an Inductive Bias for Efficient Abstractioncited by 27
  21. 2024TANGO: Training-free Embodied AI Agents for Open-world Taskscited by 27
  22. 2021Learning by Planning: Language-Guided Global Image Editingcited by 25
  23. 2021Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsulescited by 22
  24. 2021Iterated learning for emergent systematicity in VQAcited by 21
  25. 2021ProTo: Program-Guided Transformer for Program-Guided Taskscited by 21
  26. 2022FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relationscited by 20
  27. 2021ReaSCAN: Compositional Reasoning in Language Groundingcited by 19
  28. 2021Towards Language-guided Visual Recognition via Dynamic Convolutionscited by 17
  29. 2022Learning Program Representations for Food Images and Cooking Recipescited by 16
  30. 2022CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanationscited by 16
  31. 2021How Modular Should Neural Module Networks Be for Systematic Generalization?cited by 15
  32. 2021Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?cited by 15
  33. 2024Improving Generalization in Visual Reasoning via Self-Ensemblecited by 15
  34. 2021CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Imagescited by 14
  35. 2021Distribution Matching for Rationalizationcited by 14
  36. 2021Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answeringcited by 14
  37. 2021Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Imagescited by 14
  38. 2022VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challengescited by 14

50 most cited of 130 All 130 (JSON)