mlsanityData to 14 Oct 2026

Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding

arXiv 1810.02338 ↗First posted 4 Oct 2018cs.AI cs.CL cs.CVcited by 304

Not on the map. Why?

Cited by 65 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We marry two powerful ideas: deep representation learning for visual recognition and language understanding, and symbolic program execution for reasoning. Our neural-symbolic visual question answering (NS-VQA) system first recovers a structural scene representation from the image and a program trace from the question. It then executes the program on the scene representation to obtain an answer. Incorporating symbolic structure as prior knowledge offers three unique advantages. First, executing programs on a symbolic space is more robust to long program traces; our model can solve complex reasoning tasks better, achieving an accuracy of 99.8% on the CLEVR dataset. Second, the model is more data- and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering. Third, symbolic program execution offers full transparency to the reasoning process; we are thus able to interpret and diagnose each execution step.

Key facts. Cited by 304 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 61; 2022: 74; 2023: 62; 2024: 47; 2025: 32; 2026: 28.

304 papers.

Sort:

By area:

  1. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  2. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  3. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  4. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  5. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  6. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 228
  7. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  8. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  9. 2021Promises and Pitfalls of Black-Box Concept Learning Modelscited by 131
  10. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  11. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 123
  12. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 120
38 more
  1. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 119
  2. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 92
  3. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  4. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 84
  5. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  6. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  7. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  8. 2022A Neuro-vector-symbolic Architecture for Solving Raven's Progressive Matricescited by 64
  9. 2022ComPhy: Compositional Physical Reasoning of Objects and Events from Videoscited by 63
  10. 2023Scallop: A Language for Neurosymbolic Programmingcited by 63
  11. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasonerscited by 63
  12. 2022Theory of Graph Neural Networks: Representation and Learningcited by 61
  13. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  14. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 59
  15. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  16. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
  17. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  18. 2021Neuro-Symbolic Artificial Intelligence: Current Trendscited by 54
  19. 2022Leveraging Explanations in Interactive Machine Learning: An Overviewcited by 53
  20. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 51
  21. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 49
  22. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  23. 2021Leveraging Language to Learn Program Abstractions and Search Heuristicscited by 45
  24. 2021Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Executioncited by 43
  25. 2024CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solverscited by 40
  26. 2021A Survey on Neural-symbolic Learning Systemscited by 39
  27. 2022Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matricescited by 37
  28. 2022MultiViz: Towards Visualizing and Understanding Multimodal Modelscited by 36
  29. 2021DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answeringcited by 35
  30. 2022Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computingcited by 35
  31. 2024Can Large Language Models Understand Symbolic Graphics Programs?cited by 35
  32. 2021PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoningcited by 34
  33. 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33
  34. 2023What's Left? Concept Grounding with Logic-Enhanced Foundation Modelscited by 33
  35. 2022VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programmingcited by 31
  36. 2022Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?cited by 31
  37. 2021ACRE: Abstract Causal REasoning Beyond Covariationcited by 30
  38. 2021Linguistic Structures as Weak Supervision for Visual Scene Graph Generationcited by 29

50 most cited of 304 All 304 (JSON)