# Visual Graph Question Answering with ASP and LLMs for Language Parsing

arXiv: https://arxiv.org/abs/2502.09211 · HTML: https://mlsanity.com/paper/2502.09211/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 13 Feb 2025.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [A Semantic Loss Function for Deep Learning with Symbolic Knowledge](https://mlsanity.com/paper/1711.11157/) (2017)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [NeurASP: Embracing Neural Networks into Answer Set Programming](https://mlsanity.com/paper/2307.07700/) (2023)
- [ASP-Core-2 Input Language Format](https://mlsanity.com/paper/1911.04326/) (2019)
- [Leveraging Large Language Models to Generate Answer Set Programs](https://mlsanity.com/paper/2307.07699/) (2023)
- [Making sense of sensory input](https://mlsanity.com/paper/1910.02227/) (2019)
- [A Neuro-Symbolic ASP Pipeline for Visual Question Answering](https://mlsanity.com/paper/2205.07548/) (2022)
- [Reliable Natural Language Understanding with Large Language Models and Answer Set Programming](https://mlsanity.com/paper/2302.03780/) (2023)
- [Visual Question Answering: which investigated applications?](https://mlsanity.com/paper/2103.02937/) (2021)
- [Neuro-Symbolic AI for Compliance Checking of Electrical Control Panels](https://mlsanity.com/paper/2305.10113/) (2023)

## Built on by (most cited first, 2 of 2)

- [TraversalBench: Challenging Paths to Follow for Vision Language Models](https://mlsanity.com/paper/2604.10999/) (2026)
- [BLAST: Benchmarking LLMs with ASP-based Structured Testing](https://mlsanity.com/paper/2604.22306/) (2026)

## Abstract (arXiv)

Visual Question Answering (VQA) is a challenging problem that requires to process multimodal input. Answer-Set Programming (ASP) has shown great potential in this regard to add interpretability and explainability to modular VQA architectures. In this work, we address the problem of how to integrate ASP with modules for vision and natural language processing to solve a new and demanding VQA variant that is concerned with images of graphs (not graphs in symbolic form). Images containing graph-based structures are an ubiquitous and popular form of visualisation. Here, we deal with the particular problem of graphs inspired by transit networks, and we introduce a novel dataset that amends an existing one by adding images of graphs that resemble metro lines. Our modular neuro-symbolic approach combines optical graph recognition for graph parsing, a pretrained optical character recognition neural network for parsing labels, Large Language Models (LLMs) for language processing, and ASP for reasoning. This method serves as a first baseline and achieves an overall average accuracy of 73% on the dataset. Our evaluation provides further evidence of the potential of modular neuro-symbolic systems, in particular with pretrained models that do not involve any further training and logic programming for reasoning, to solve complex VQA tasks.