# The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions

arXiv: https://arxiv.org/abs/1612.05386 · HTML: https://mlsanity.com/paper/1612.05386/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Dec 2016.

## Built on by (most cited first, 10 of 10)

- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [A Comprehensive Survey on Visual Question Answering Datasets and Algorithms](https://mlsanity.com/paper/2411.11150/) (2024)
- [From Pixels to Objects: Cubic Visual Attention for Visual Question Answering](https://mlsanity.com/paper/2206.01923/) (2022)
- [MuVAM: A Multi-View Attention-based Model for Medical Visual Question Answering](https://mlsanity.com/paper/2107.03216/) (2021)
- [Divide and Conquer: Answering Questions with Object Factorization and Compositional Reasoning](https://mlsanity.com/paper/2303.10482/) (2023)
- [LadRa-Net: Locally-Aware Dynamic Re-read Attention Net for Sentence Semantic Matching](https://mlsanity.com/paper/2108.02915/) (2021)
- [Multimodal E-Commerce Product Classification Using Hierarchical Fusion](https://mlsanity.com/paper/2207.03305/) (2022)

## Abstract (arXiv)

One of the most intriguing features of the Visual Question Answering (VQA) challenge is the unpredictability of the questions. Extracting the information required to answer them demands a variety of image operations from detection and counting, to segmentation and reconstruction. To train a method to perform even one of these operations accurately from {image,question,answer} tuples would be challenging, but to aim to achieve them all with a limited set of such training data seems ambitious at best. We propose here instead a more general and scalable approach which exploits the fact that very good methods to achieve these operations already exist, and thus do not need to be trained. Our method thus learns how to exploit a set of external off-the-shelf algorithms to achieve its goal, an approach that has something in common with the Neural Turing Machine. The core of our proposed method is a new co-attention model. In addition, the proposed approach generates human-readable reasons for its decision, and can still be trained end-to-end without ground truth reasons being given. We demonstrate the effectiveness on two publicly available datasets, Visual Genome and VQA, and show that it produces the state-of-the-art results in both cases.