# Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention

arXiv: https://arxiv.org/abs/1706.08474 · HTML: https://mlsanity.com/paper/1706.08474/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2017.

## Built on by (most cited first, 10 of 10)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Deep Saliency Prior for Reducing Visual Distraction](https://mlsanity.com/paper/2109.01980/) (2021)
- [Trends, Applications, and Challenges in Human Attention Modelling](https://mlsanity.com/paper/2402.18673/) (2024)
- [Multi-Modal Image Captioning for the Visually Impaired](https://mlsanity.com/paper/2105.08106/) (2021)
- [Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting](https://mlsanity.com/paper/2404.12782/) (2024)
- [Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training](https://mlsanity.com/paper/2201.04026/) (2022)
- [Learning to Select: A Fully Attentive Approach for Novel Object Captioning](https://mlsanity.com/paper/2106.01424/) (2021)
- [Teaching Human Behavior Improves Content Understanding Abilities Of LLMs](https://mlsanity.com/paper/2405.00942/) (2024)
- [Personalized Image Descriptions from Attention Sequences](https://mlsanity.com/paper/2512.06662/) (2025)
- [A look under the hood of the Interactive Deep Learning Enterprise (No-IDLE)](https://mlsanity.com/paper/2406.19054/) (2024)

## Abstract (arXiv)

Image captioning has been recently gaining a lot of attention thanks to the impressive achievements shown by deep captioning architectures, which combine Convolutional Neural Networks to extract image representations, and Recurrent Neural Networks to generate the corresponding captions. At the same time, a significant research effort has been dedicated to the development of saliency prediction models, which can predict human eye fixations. Even though saliency information could be useful to condition an image captioning architecture, by providing an indication of what is salient and what is not, research is still struggling to incorporate these two techniques. In this work, we propose an image captioning approach in which a generative recurrent neural network can focus on different parts of the input image during the generation of the caption, by exploiting the conditioning given by a saliency prediction model on which parts of the image are salient and which are contextual. We show, through extensive quantitative and qualitative experiments on large scale datasets, that our model achieves superior performances with respect to captioning baselines with and without saliency, and to different state of the art approaches combining saliency and captioning.