# R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering

arXiv: https://arxiv.org/abs/1805.09701 · HTML: https://mlsanity.com/paper/1805.09701/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2018.

## Built on by (most cited first, 23 of 23)

- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning](https://mlsanity.com/paper/2110.13214/) (2021)
- [Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models](https://mlsanity.com/paper/2310.05338/) (2023)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram](https://mlsanity.com/paper/2306.15681/) (2023)
- [Coarse-to-Fine Reasoning for Visual Question Answering](https://mlsanity.com/paper/2110.02526/) (2021)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [Modern Question Answering Datasets and Benchmarks: A Survey](https://mlsanity.com/paper/2206.15030/) (2022)
- [IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web](https://mlsanity.com/paper/2409.18980/) (2024)
- [A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answering](https://mlsanity.com/paper/2201.05299/) (2022)
- [A survey on VQA_Datasets and Approaches](https://mlsanity.com/paper/2105.00421/) (2021)
- [Structured Two-stream Attention Network for Video Question Answering](https://mlsanity.com/paper/2206.01017/) (2022)
- [Combo of Thinking and Observing for Outside-Knowledge VQA](https://mlsanity.com/paper/2305.06407/) (2023)
- [GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2402.02503/) (2024)
- [Exploring Semantic Relationships for Unpaired Image Captioning](https://mlsanity.com/paper/2106.10658/) (2021)
- [Bilateral Cross-Modality Graph Matching Attention for Feature Fusion in Visual Question Answering](https://mlsanity.com/paper/2112.07270/) (2021)
- [NEWSKVQA: Knowledge-Aware News Video Question Answering](https://mlsanity.com/paper/2202.04015/) (2022)
- [Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trends](https://mlsanity.com/paper/2401.07518/) (2024)
- [COCO is "ALL'' You Need for Visual Instruction Fine-tuning](https://mlsanity.com/paper/2401.08968/) (2024)
- [FedMobile: Enabling Knowledge Contribution-aware Multi-modal Federated Learning with Incomplete Modalities](https://mlsanity.com/paper/2502.15839/) (2025)
- [Visual Question Answering based on Local-Scene-Aware Referring Expression Generation](https://mlsanity.com/paper/2101.08978/) (2021)
- [Multi-Clue Reasoning with Memory Augmentation for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2312.12723/) (2023)
- [A Framework for Generating Artificial Datasets to Validate Absolute and Relative Position Concepts](https://mlsanity.com/paper/2509.18177/) (2025)

## Abstract (arXiv)

Recently, Visual Question Answering (VQA) has emerged as one of the most significant tasks in multimodal learning as it requires understanding both visual and textual modalities. Existing methods mainly rely on extracting image and question features to learn their joint feature embedding via multimodal fusion or attention mechanism. Some recent studies utilize external VQA-independent models to detect candidate entities or attributes in images, which serve as semantic knowledge complementary to the VQA task. However, these candidate entities or attributes might be unrelated to the VQA task and have limited semantic capacities. To better utilize semantic knowledge in images, we propose a novel framework to learn visual relation facts for VQA. Specifically, we build up a Relation-VQA (R-VQA) dataset based on the Visual Genome dataset via a semantic similarity module, in which each data consists of an image, a corresponding question, a correct answer and a supporting relation fact. A well-defined relation detector is then adopted to predict visual question-related relation facts. We further propose a multi-step attention model composed of visual attention and semantic attention sequentially to extract related visual knowledge and semantic knowledge. We conduct comprehensive experiments on the two benchmark datasets, demonstrating that our model achieves state-of-the-art performance and verifying the benefit of considering visual relation facts.