# Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge

arXiv: https://arxiv.org/abs/1708.02711 · HTML: https://mlsanity.com/paper/1708.02711/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Aug 2017.

## Built on by (most cited first, 30 of 76)

- [Scaling Language-Image Pre-training via Masking](https://mlsanity.com/paper/2212.00794/) (2022)
- [GLIPv2: Unifying Localization and Vision-Language Understanding](https://mlsanity.com/paper/2206.05836/) (2022)
- [Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2108.04024/) (2021)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly](https://mlsanity.com/paper/2204.13631/) (2022)
- [Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?](https://mlsanity.com/paper/2112.13906/) (2021)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers](https://mlsanity.com/paper/2102.00529/) (2021)
- [Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering](https://mlsanity.com/paper/2107.02331/) (2021)
- [BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning](https://mlsanity.com/paper/2206.08657/) (2022)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Data](https://mlsanity.com/paper/2205.03147/) (2022)
- [Dual-Key Multimodal Backdoors for Visual Question Answering](https://mlsanity.com/paper/2112.07668/) (2021)
- [(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering](https://mlsanity.com/paper/2202.09277/) (2022)
- [Probabilistic Compositional Embeddings for Multimodal Image Retrieval](https://mlsanity.com/paper/2204.05845/) (2022)
- [Contextual Dropout: An Efficient Sample-Dependent Dropout Module](https://mlsanity.com/paper/2103.04181/) (2021)
- [Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning](https://mlsanity.com/paper/2207.02598/) (2022)
- [Bayesian Attention Belief Networks](https://mlsanity.com/paper/2106.05251/) (2021)
- [Coarse-to-Fine Reasoning for Visual Question Answering](https://mlsanity.com/paper/2110.02526/) (2021)
- [Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baseline](https://mlsanity.com/paper/2209.12028/) (2022)
- [Improving Selective Visual Question Answering by Learning from Your Peers](https://mlsanity.com/paper/2306.08751/) (2023)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [Separating Skills and Concepts for Novel Visual Question Answering](https://mlsanity.com/paper/2107.09106/) (2021)
- [Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules](https://mlsanity.com/paper/2105.04836/) (2021)
- [Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literature](https://mlsanity.com/paper/2305.11033/) (2023)
- [OpenViVQA: Task, Dataset, and Multimodal Fusion Models for Visual Question Answering in Vietnamese](https://mlsanity.com/paper/2305.04183/) (2023)
- [ICDAR 2021 Competition on Document VisualQuestion Answering](https://mlsanity.com/paper/2111.05547/) (2021)
- [Learning to Retrieve Videos by Asking Questions](https://mlsanity.com/paper/2205.05739/) (2022)
- [LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injection](https://mlsanity.com/paper/2207.12888/) (2022)

## Abstract (arXiv)

This paper presents a state-of-the-art model for visual question answering (VQA), which won the first place in the 2017 VQA Challenge. VQA is a task of significant importance for research in artificial intelligence, given its multimodal nature, clear evaluation protocol, and potential real-world applications. The performance of deep neural networks for VQA is very dependent on choices of architectures and hyperparameters. To help further research in the area, we describe in detail our high-performing, though relatively simple model. Through a massive exploration of architectures and hyperparameters representing more than 3,000 GPU-hours, we identified tips and tricks that lead to its success, namely: sigmoid outputs, soft training targets, image features from bottom-up attention, gated tanh activations, output embeddings initialized using GloVe and Google Images, large mini-batches, and smart shuffling of training data. We provide a detailed analysis of their impact on performance to assist others in making an appropriate selection.