# Learning Visual Question Answering by Bootstrapping Hard Attention

arXiv: https://arxiv.org/abs/1808.00300 · HTML: https://mlsanity.com/paper/1808.00300/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Aug 2018.

## Built on by (most cited first, 24 of 24)

- [A General Survey on Attention Mechanisms in Deep Learning](https://mlsanity.com/paper/2203.14263/) (2022)
- [FANet: A Feedback Attention Network for Improved Biomedical Image Segmentation](https://mlsanity.com/paper/2103.17235/) (2021)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [Variable-rate discrete representation learning](https://mlsanity.com/paper/2103.06089/) (2021)
- [Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoning](https://mlsanity.com/paper/2106.03921/) (2021)
- [AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Loss](https://mlsanity.com/paper/2105.01993/) (2021)
- [What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memes](https://mlsanity.com/paper/2212.00715/) (2022)
- [AOE-Net: Entities Interactions Modeling with Adaptive Attention Mechanism for Temporal Action Proposals Generation](https://mlsanity.com/paper/2210.02578/) (2022)
- [MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing](https://mlsanity.com/paper/2405.11215/) (2024)
- [MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization](https://mlsanity.com/paper/2305.15913/) (2023)
- [Answer Questions with Right Image Regions: A Visual Attention Regularization Approach](https://mlsanity.com/paper/2102.01916/) (2021)
- [Co-VQA : Answering by Interactive Sub Question Sequence](https://mlsanity.com/paper/2204.00879/) (2022)
- [Unifying Global and Local Scene Entities Modelling for Precise Action Spotting](https://mlsanity.com/paper/2404.09951/) (2024)
- [Video Violence Recognition and Localization Using a Semi-Supervised Hard Attention Model](https://mlsanity.com/paper/2202.02212/) (2022)
- [Contextual Explainable Video Representation: Human Perception-based Understanding](https://mlsanity.com/paper/2212.06206/) (2022)
- [GQHAN: A Grover-inspired Quantum Hard Attention Network](https://mlsanity.com/paper/2401.14089/) (2024)
- [A Spatio-Temporal Attentive Network for Video-Based Crowd Counting](https://mlsanity.com/paper/2208.11339/) (2022)
- [SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memes](https://mlsanity.com/paper/2412.20541/) (2024)
- [A Picture May Be Worth a Hundred Words for Visual Question Answering](https://mlsanity.com/paper/2106.13445/) (2021)
- [Measuring CLEVRness: Blackbox testing of Visual Reasoning Models](https://mlsanity.com/paper/2202.12162/) (2022)
- [Improved RAMEN: Towards Domain Generalization for Visual Question Answering](https://mlsanity.com/paper/2109.02370/) (2021)
- [Fair Inference for Discrete Latent Variable Models](https://mlsanity.com/paper/2209.07044/) (2022)
- [Towards Self-Explainable Document Visual Question Answering with Chain-of-Explanation Predictions](https://mlsanity.com/paper/2605.06058/) (2026)
- [CL-DMDF:Dynamic Multimodal Data Fusion Model Based on Contrastive Learning](https://mlsanity.com/paper/2606.02659/) (2026)

## Abstract (arXiv)

Attention mechanisms in biological perception are thought to select subsets of perceptual information for more sophisticated processing which would be prohibitive to perform on all sensory inputs. In computer vision, however, there has been relatively little exploration of hard attention, where some information is selectively ignored, in spite of the success of soft attention, where information is re-weighted and aggregated, but never filtered out. Here, we introduce a new approach for hard attention and find it achieves very competitive performance on a recently-released visual question answering datasets, equalling and in some cases surpassing similar soft attention architectures while entirely ignoring some features. Even though the hard attention mechanism is thought to be non-differentiable, we found that the feature magnitudes correlate with semantic relevance, and provide a useful signal for our mechanism's attentional selection criterion. Because hard attention selects important features of the input information, it can also be more efficient than analogous soft attention mechanisms. This is especially important for recent approaches that use non-local pairwise operations, whereby computational and memory costs are quadratic in the size of the set of features.