# LRTA: A Transparent Neural-Symbolic Reasoning Framework with Modular Supervision for Visual Question Answering

arXiv: https://arxiv.org/abs/2011.10731 · HTML: https://mlsanity.com/paper/2011.10731/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2020.

## Built on by (most cited first, 12 of 12)

- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answering](https://mlsanity.com/paper/2104.10283/) (2021)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations](https://mlsanity.com/paper/2106.00162/) (2021)
- [Neuro-Symbolic Learning: Principles and Applications in Ophthalmology](https://mlsanity.com/paper/2208.00374/) (2022)
- [MGA-VQA: Multi-Granularity Alignment for Visual Question Answering](https://mlsanity.com/paper/2201.10656/) (2022)
- [Towards Reasoning-Aware Explainable VQA](https://mlsanity.com/paper/2211.05190/) (2022)
- [SelfGraphVQA: A Self-Supervised Graph Neural Network for Scene-based Question Answering](https://mlsanity.com/paper/2310.01842/) (2023)
- [Declarative Knowledge Distillation from Large Language Models for Visual Question Answering Datasets](https://mlsanity.com/paper/2410.09428/) (2024)
- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)
- [SA-VQA: Structured Alignment of Visual and Semantic Representations for Visual Question Answering](https://mlsanity.com/paper/2201.10654/) (2022)
- [A Multi-level Alignment Training Scheme for Video-and-Language Grounding](https://mlsanity.com/paper/2204.10938/) (2022)

## Abstract (arXiv)

The predominant approach to visual question answering (VQA) relies on encoding the image and question with a "black-box" neural encoder and decoding a single token as the answer like "yes" or "no". Despite this approach's strong quantitative results, it struggles to come up with intuitive, human-readable forms of justification for the prediction process. To address this insufficiency, we reformulate VQA as a full answer generation task, which requires the model to justify its predictions in natural language. We propose LRTA [Look, Read, Think, Answer], a transparent neural-symbolic reasoning framework for visual question answering that solves the problem step-by-step like humans and provides human-readable form of justification at each step. Specifically, LRTA learns to first convert an image into a scene graph and parse a question into multiple reasoning instructions. It then executes the reasoning instructions one at a time by traversing the scene graph using a recurrent neural-symbolic execution module. Finally, it generates a full answer to the given question with natural language justifications. Our experiments on GQA dataset show that LRTA outperforms the state-of-the-art model by a large margin (43.1% v.s. 28.0%) on the full answer generation task. We also create a perturbed GQA test set by removing linguistic cues (attributes and relations) in the questions for analyzing whether a model is having a smart guess with superficial data correlations. We show that LRTA makes a step towards truly understanding the question while the state-of-the-art model tends to learn superficial correlations from the training data.