# A Neuro-Symbolic ASP Pipeline for Visual Question Answering

arXiv: https://arxiv.org/abs/2205.07548 · HTML: https://mlsanity.com/paper/2205.07548/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 16 May 2022.

## Builds on (most cited first, 13 of 13)

- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Neural Probabilistic Logic Programming in DeepProbLog](https://mlsanity.com/paper/1907.08194/) (2019)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [A Semantic Loss Function for Deep Learning with Symbolic Knowledge](https://mlsanity.com/paper/1711.11157/) (2017)
- [Exploring Models and Data for Image Question Answering](https://mlsanity.com/paper/1505.02074/) (2015)
- [A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input](https://mlsanity.com/paper/1410.0210/) (2014)
- [CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images](https://mlsanity.com/paper/2104.05981/) (2021)

## Built on by (most cited first, 14 of 14)

- [Scalable Neural-Probabilistic Answer Set Programming](https://mlsanity.com/paper/2306.08397/) (2023)
- [The Role of Foundation Models in Neuro-Symbolic Learning and Reasoning](https://mlsanity.com/paper/2402.01889/) (2024)
- [Dspy-based Neural-Symbolic Pipeline to Enhance Spatial Reasoning in LLMs](https://mlsanity.com/paper/2411.18564/) (2024)
- [Differentiable Logic Programming for Distant Supervision](https://mlsanity.com/paper/2408.12591/) (2024)
- [Visual Graph Question Answering with ASP and LLMs for Language Parsing](https://mlsanity.com/paper/2502.09211/) (2025)
- [Towards Commonsense Knowledge based Fuzzy Systems for Supporting Size-Related Fine-Grained Object Detection](https://mlsanity.com/paper/2303.09026/) (2023)
- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)
- [Neural Decision-Propagation for Answer Set Programming](https://mlsanity.com/paper/2605.01797/) (2026)
- [HIVA: Holographic Intellectual Voice Assistant](https://mlsanity.com/paper/2307.05501/) (2023)
- [Assistive Image Annotation Systems with Deep Learning and Natural Language Capabilities: A Review](https://mlsanity.com/paper/2407.00252/) (2024)
- [Abduction of Domain Relationships from Data for VQA](https://mlsanity.com/paper/2502.09219/) (2025)
- [The Dual Role of Abstracting over the Irrelevant in Symbolic Explanations: Cognitive Effort vs. Understanding](https://mlsanity.com/paper/2602.03467/) (2026)
- [Answer Set Programming Energised! End-to-End Neurosymbolic Reasoning and Learning with ASP and Energy Based Models](https://mlsanity.com/paper/2607.08136/) (2026)
- [Distilling Answer Set Programming Theories from Large Language Models](https://mlsanity.com/paper/2607.28086/) (2026)

## Abstract (arXiv)

We present a neuro-symbolic visual question answering (VQA) pipeline for CLEVR, which is a well-known dataset that consists of pictures showing scenes with objects and questions related to them. Our pipeline covers (i) training neural networks for object classification and bounding-box prediction of the CLEVR scenes, (ii) statistical analysis on the distribution of prediction values of the neural networks to determine a threshold for high-confidence predictions, and (iii) a translation of CLEVR questions and network predictions that pass confidence thresholds into logic programs so that we can compute the answers using an ASP solver. By exploiting choice rules, we consider deterministic and non-deterministic scene encodings. Our experiments show that the non-deterministic scene encoding achieves good results even if the neural networks are trained rather poorly in comparison with the deterministic approach. This is important for building robust VQA systems if network predictions are less-than perfect. Furthermore, we show that restricting non-determinism to reasonable choices allows for more efficient implementations in comparison with related neuro-symbolic approaches without loosing much accuracy. This work is under consideration for acceptance in TPLP.