# Boosted Attention: Leveraging Human Attention for Image Captioning

arXiv: https://arxiv.org/abs/1904.00767 · HTML: https://mlsanity.com/paper/1904.00767/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Mar 2019.

## Built on by (most cited first, 8 of 8)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Visual Attention Methods in Deep Learning: An In-Depth Survey](https://mlsanity.com/paper/2204.07756/) (2022)
- [Neural Attention for Image Captioning: Review of Outstanding Methods](https://mlsanity.com/paper/2111.15015/) (2021)
- [Contrastive Language-Image Pretrained Models are Zero-Shot Human Scanpath Predictors](https://mlsanity.com/paper/2305.12380/) (2023)
- [Personalized Image Descriptions from Attention Sequences](https://mlsanity.com/paper/2512.06662/) (2025)
- [Cross Modification Attention Based Deliberation Model for Image Captioning](https://mlsanity.com/paper/2109.08411/) (2021)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Learning to Refer from Estimated Listener Gaze](https://mlsanity.com/paper/2609.14207/) (2026)

## Abstract (arXiv)

Visual attention has shown usefulness in image captioning, with the goal of enabling a caption model to selectively focus on regions of interest. Existing models typically rely on top-down language information and learn attention implicitly by optimizing the captioning objectives. While somewhat effective, the learned top-down attention can fail to focus on correct regions of interest without direct supervision of attention. Inspired by the human visual system which is driven by not only the task-specific top-down signals but also the visual stimuli, we in this work propose to use both types of attention for image captioning. In particular, we highlight the complementary nature of the two types of attention and develop a model (Boosted Attention) to integrate them for image captioning. We validate the proposed approach with state-of-the-art performance across various evaluation metrics.