# Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning"

arXiv: https://arxiv.org/abs/2006.11524 · HTML: https://mlsanity.com/paper/2006.11524/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2020.

## Built on by (most cited first, 30 of 35)

- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Disentangled Representation Learning](https://mlsanity.com/paper/2211.11695/) (2022)
- [Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning](https://mlsanity.com/paper/2103.16564/) (2021)
- [Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language](https://mlsanity.com/paper/2110.15358/) (2021)
- [Neuro-Symbolic Artificial Intelligence: Current Trends](https://mlsanity.com/paper/2105.05330/) (2021)
- [Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computing](https://mlsanity.com/paper/2210.15889/) (2022)
- [VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search](https://mlsanity.com/paper/2503.10582/) (2025)
- [Coarse-to-Fine Reasoning for Visual Question Answering](https://mlsanity.com/paper/2110.02526/) (2021)
- [Neuro-Symbolic Hierarchical Rule Induction](https://mlsanity.com/paper/2112.13418/) (2021)
- [GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answering](https://mlsanity.com/paper/2104.10283/) (2021)
- [Neuro-Symbolic Forward Reasoning](https://mlsanity.com/paper/2110.09383/) (2021)
- [Visual Knowledge in the Big Model Era: Retrospect and Prospect](https://mlsanity.com/paper/2404.04308/) (2024)
- [Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images](https://mlsanity.com/paper/2110.00519/) (2021)
- [Trajectory Prediction with Linguistic Representations](https://mlsanity.com/paper/2110.09741/) (2021)
- [Domain-robust VQA with diverse datasets and methods but no target labels](https://mlsanity.com/paper/2103.15974/) (2021)
- [Truth-Conditional Captioning of Time Series Data](https://mlsanity.com/paper/2110.01839/) (2021)
- [Building Intelligent Agents with Neuro-Symbolic Concepts](https://mlsanity.com/paper/2505.06191/) (2025)
- [DeiSAM: Segment Anything with Deictic Prompting](https://mlsanity.com/paper/2402.14123/) (2024)
- [Measuring Faithful and Plausible Visual Grounding in VQA](https://mlsanity.com/paper/2305.15015/) (2023)
- [From Pixels to Graphs: using Scene and Knowledge Graphs for HD-EPIC VQA Challenge](https://mlsanity.com/paper/2506.08553/) (2025)
- [On the Role of Visual Grounding in VQA](https://mlsanity.com/paper/2406.18253/) (2024)
- [A Study on Neuro-Symbolic Artificial Intelligence: Healthcare Perspectives](https://mlsanity.com/paper/2503.18213/) (2025)
- [LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering](https://mlsanity.com/paper/2111.14547/) (2021)
- [Rapid Image Labeling via Neuro-Symbolic Learning](https://mlsanity.com/paper/2306.10490/) (2023)
- [Visually Grounded VQA by Lattice-based Retrieval](https://mlsanity.com/paper/2211.08086/) (2022)
- [HySTER: A Hybrid Spatio-Temporal Event Reasoner](https://mlsanity.com/paper/2101.06644/) (2021)
- [Data augmentation by morphological mixup for solving Raven's Progressive Matrices](https://mlsanity.com/paper/2103.05222/) (2021)
- [Semantically Adversarial Scenario Generation with Explicit Knowledge Guidance](https://mlsanity.com/paper/2106.04066/) (2021)
- [A Stepwise Distillation Learning Strategy for Non-differentiable Visual Programming Frameworks on Visual Reasoning Tasks](https://mlsanity.com/paper/2309.09809/) (2023)
- [Enhancing Visual Programming for Visual Reasoning via Probabilistic Graphs](https://mlsanity.com/paper/2512.14257/) (2025)

## Abstract (arXiv)

Visual reasoning tasks such as visual question answering (VQA) require an interplay of visual perception with reasoning about the question semantics grounded in perception. However, recent advances in this area are still primarily driven by perception improvements (e.g. scene graph generation) rather than reasoning. Neuro-symbolic models such as Neural Module Networks bring the benefits of compositional reasoning to VQA, but they are still entangled with visual representation learning, and thus neural reasoning is hard to improve and assess on its own. To address this, we propose (1) a framework to isolate and evaluate the reasoning aspect of VQA separately from its perception, and (2) a novel top-down calibration technique that allows the model to answer reasoning questions even with imperfect perception. To this end, we introduce a differentiable first-order logic formalism for VQA that explicitly decouples question answering from visual perception. On the challenging GQA dataset, this framework is used to perform in-depth, disentangled comparisons between well-known VQA models leading to informative insights regarding the participating models as well as the task.