# Sparse and Structured Visual Attention

arXiv: https://arxiv.org/abs/2002.05556 · HTML: https://mlsanity.com/paper/2002.05556/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2020.

## Built on by (most cited first, 5 of 5)

- [Towards Self-Supervised Learning of Global and Object-Centric Representations](https://mlsanity.com/paper/2203.05997/) (2022)
- [Multimodal Continuous Visual Attention Mechanisms](https://mlsanity.com/paper/2104.03046/) (2021)
- [Reranking Laws for Language Generation: A Communication-Theoretic Perspective](https://mlsanity.com/paper/2409.07131/) (2024)
- [Key Patches Are All You Need: A Multiple Instance Learning Framework For Robust Medical Diagnosis](https://mlsanity.com/paper/2405.01654/) (2024)
- [Beyond Flat Unknown Labels in Open-World Object Detection](https://mlsanity.com/paper/2510.09173/) (2025)

## Abstract (arXiv)

Visual attention mechanisms are widely used in multimodal tasks, as visual question answering (VQA). One drawback of softmax-based attention mechanisms is that they assign some probability mass to all image regions, regardless of their adjacency structure and of their relevance to the text. In this paper, to better link the image structure with the text, we replace the traditional softmax attention mechanism with two alternative sparsity-promoting transformations: sparsemax, which is able to select only the relevant regions (assigning zero weight to the rest), and a newly proposed Total-Variation Sparse Attention (TVmax), which further encourages the joint selection of adjacent spatial locations. Experiments in VQA show gains in accuracy as well as higher similarity to human attention, which suggests better interpretability.