# Relation-Aware Graph Attention Network for Visual Question Answering

arXiv: https://arxiv.org/abs/1903.12314 · HTML: https://mlsanity.com/paper/1903.12314/

## Key facts

- Cited by 119 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2019.

## Built on by (most cited first, 30 of 119)

- [Scaling Local Self-Attention for Parameter Efficient Visual Backbones](https://mlsanity.com/paper/2103.12731/) (2021)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning](https://mlsanity.com/paper/2212.00259/) (2022)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models](https://mlsanity.com/paper/2106.00245/) (2021)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Video as Conditional Graph Hierarchy for Multi-Granular Question Answering](https://mlsanity.com/paper/2112.06197/) (2021)
- [Beyond One-to-One: Rethinking the Referring Image Segmentation](https://mlsanity.com/paper/2308.13853/) (2023)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [EGTR: Extracting Graph from Transformer for Scene Graph Generation](https://mlsanity.com/paper/2404.02072/) (2024)
- [Act As You Wish: Fine-Grained Control of Motion Diffusion Model with Hierarchical Semantic Graphs](https://mlsanity.com/paper/2311.01015/) (2023)
- [Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering](https://mlsanity.com/paper/2307.11986/) (2023)
- [DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering](https://mlsanity.com/paper/2107.04768/) (2021)
- [Multimodal Dialogue Response Generation](https://mlsanity.com/paper/2110.08515/) (2021)
- [ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration](https://mlsanity.com/paper/2108.07073/) (2021)
- [STAR: A First-Ever Dataset and A Large-Scale Benchmark for Scene Graph Generation in Large-Size Satellite Imagery](https://mlsanity.com/paper/2406.09410/) (2024)
- [(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering](https://mlsanity.com/paper/2202.09277/) (2022)
- [Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models](https://mlsanity.com/paper/2408.09429/) (2024)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [Coarse-to-Fine Reasoning for Visual Question Answering](https://mlsanity.com/paper/2110.02526/) (2021)
- [Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answering](https://mlsanity.com/paper/2209.10326/) (2022)
- [Robust Visual Question Answering: Datasets, Methods, and Future Challenges](https://mlsanity.com/paper/2307.11471/) (2023)
- [SPA: A Graph Spectral Alignment Perspective for Domain Adaptation](https://mlsanity.com/paper/2310.17594/) (2023)
- [Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2303.01903/) (2023)
- [GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinement](https://mlsanity.com/paper/2208.08965/) (2022)
- [Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem](https://mlsanity.com/paper/2207.11850/) (2022)
- [Efficient Graph Deep Learning in TensorFlow with tf_geometric](https://mlsanity.com/paper/2101.11552/) (2021)
- [Structured Co-reference Graph Attention for Video-grounded Dialogue](https://mlsanity.com/paper/2103.13361/) (2021)

## Abstract (arXiv)

In order to answer semantically-complicated questions about an image, a Visual Question Answering (VQA) model needs to fully understand the visual scene in the image, especially the interactive dynamics between different objects. We propose a Relation-aware Graph Attention Network (ReGAT), which encodes each image into a graph and models multi-type inter-object relations via a graph attention mechanism, to learn question-adaptive relation representations. Two types of visual object relations are explored: (i) Explicit Relations that represent geometric positions and semantic interactions between objects; and (ii) Implicit Relations that capture the hidden dynamics between image regions. Experiments demonstrate that ReGAT outperforms prior state-of-the-art approaches on both VQA 2.0 and VQA-CP v2 datasets. We further show that ReGAT is compatible to existing VQA architectures, and can be used as a generic relation encoder to boost the model performance for VQA.