# Coarse-to-Fine Reasoning for Visual Question Answering

arXiv: https://arxiv.org/abs/2110.02526 · HTML: https://mlsanity.com/paper/2110.02526/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 70 references that are in the map.
- First posted on arXiv 6 Oct 2021.

## Builds on (most cited first, 30 of 54)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [FVQA: Fact-based Visual Question Answering](https://mlsanity.com/paper/1606.05433/) (2016)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [IQA: Visual Question Answering in Interactive Environments](https://mlsanity.com/paper/1712.03316/) (2017)
- [In Defense of Grid Features for Visual Question Answering](https://mlsanity.com/paper/2001.03615/) (2020)
- [AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection](https://mlsanity.com/paper/1709.07326/) (2017)
- [Graph-Structured Representations for Visual Question Answering](https://mlsanity.com/paper/1609.05600/) (2016)
- [Relation-Aware Graph Attention Network for Visual Question Answering](https://mlsanity.com/paper/1903.12314/) (2019)
- [Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering](https://mlsanity.com/paper/1812.05252/) (2018)
- [Learning by Abstraction: The Neural State Machine](https://mlsanity.com/paper/1907.03950/) (2019)
- [Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networks](https://mlsanity.com/paper/1812.04794/) (2018)
- [Scene Graph Generation with External Knowledge and Image Reconstruction](https://mlsanity.com/paper/1904.00560/) (2019)
- [Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge](https://mlsanity.com/paper/1708.02711/) (2017)
- [Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models](https://mlsanity.com/paper/2106.00245/) (2021)

## Built on by (most cited first, 25 of 25)

- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2306.09265/) (2023)
- [MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning](https://mlsanity.com/paper/2112.05253/) (2021)
- [Deep Federated Learning for Autonomous Driving](https://mlsanity.com/paper/2110.05754/) (2021)
- [Robust Visual Question Answering: Datasets, Methods, and Future Challenges](https://mlsanity.com/paper/2307.11471/) (2023)
- [VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering](https://mlsanity.com/paper/2205.11501/) (2022)
- [Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literature](https://mlsanity.com/paper/2305.11033/) (2023)
- [SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge](https://mlsanity.com/paper/2405.14554/) (2024)
- [Enhancing Visual Question Answering through Ranking-Based Hybrid Training and Multimodal Fusion](https://mlsanity.com/paper/2408.07303/) (2024)
- [Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmers](https://mlsanity.com/paper/2401.01974/) (2024)
- [Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering](https://mlsanity.com/paper/2311.17331/) (2023)
- [Fine-Grained Visual Classification using Self Assessment Classifier](https://mlsanity.com/paper/2205.10529/) (2022)
- [FlexCap: Describe Anything in Images in Controllable Detail](https://mlsanity.com/paper/2403.12026/) (2024)
- [SADL: An Effective In-Context Learning Method for Compositional Visual QA](https://mlsanity.com/paper/2407.01983/) (2024)
- [A Survey on Image-text Multimodal Models](https://mlsanity.com/paper/2309.15857/) (2023)
- [Towards Reasoning-Aware Explainable VQA](https://mlsanity.com/paper/2211.05190/) (2022)
- [A Survey on Interpretable Cross-modal Reasoning](https://mlsanity.com/paper/2309.01955/) (2023)
- [SelfGraphVQA: A Self-Supervised Graph Neural Network for Scene-based Question Answering](https://mlsanity.com/paper/2310.01842/) (2023)
- [A Stepwise Distillation Learning Strategy for Non-differentiable Visual Programming Frameworks on Visual Reasoning Tasks](https://mlsanity.com/paper/2309.09809/) (2023)
- [Predicate Hierarchies Improve Few-Shot State Classification](https://mlsanity.com/paper/2502.12481/) (2025)
- [Efficiently Enhancing General Agents With Hierarchical-categorical Memory](https://mlsanity.com/paper/2505.22006/) (2025)
- [Physics-Based Benchmarking Metrics for Multimodal Synthetic Images](https://mlsanity.com/paper/2511.15204/) (2025)
- [Joint Adaptive Representations for Image-Language Learning](https://mlsanity.com/paper/2305.19924/) (2023)
- [Multi-Sourced Compositional Generalization in Visual Question Answering](https://mlsanity.com/paper/2505.23045/) (2025)
- [A Framework for Generating Artificial Datasets to Validate Absolute and Relative Position Concepts](https://mlsanity.com/paper/2509.18177/) (2025)

## Abstract (arXiv)

Bridging the semantic gap between image and question is an important step to improve the accuracy of the Visual Question Answering (VQA) task. However, most of the existing VQA methods focus on attention mechanisms or visual relations for reasoning the answer, while the features at different semantic levels are not fully utilized. In this paper, we present a new reasoning framework to fill the gap between visual features and semantic clues in the VQA task. Our method first extracts the features and predicates from the image and question. We then propose a new reasoning framework to effectively jointly learn these features and predicates in a coarse-to-fine manner. The intensively experimental results on three large-scale VQA datasets show that our proposed approach achieves superior accuracy comparing with other state-of-the-art methods. Furthermore, our reasoning framework also provides an explainable way to understand the decision of the deep neural network when predicting the answer.