# Areas of Attention for Image Captioning

arXiv: https://arxiv.org/abs/1612.01033 · HTML: https://mlsanity.com/paper/1612.01033/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2016.

## Built on by (most cited first, 24 of 24)

- [Generalizing Face Forgery Detection with High-frequency Features](https://mlsanity.com/paper/2103.12376/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [M^4I: Multi-modal Models Membership Inference](https://mlsanity.com/paper/2209.06997/) (2022)
- [CIDEr-R: Robust Consensus-based Image Description Evaluation](https://mlsanity.com/paper/2109.13701/) (2021)
- [PIXEL: Physics-Informed Cell Representations for Fast and Accurate PDE Solvers](https://mlsanity.com/paper/2207.12800/) (2022)
- [Neural Attention for Image Captioning: Review of Outstanding Methods](https://mlsanity.com/paper/2111.15015/) (2021)
- [A Novel Evaluation Framework for Image2Text Generation](https://mlsanity.com/paper/2408.01723/) (2024)
- [Visual Question Answering based on Local-Scene-Aware Referring Expression Generation](https://mlsanity.com/paper/2101.08978/) (2021)
- [Desk Organization: Effect of Multimodal Inputs on Spatial Relational Learning](https://mlsanity.com/paper/2108.01254/) (2021)
- [Are metrics measuring what they should? An evaluation of image captioning task metrics](https://mlsanity.com/paper/2207.01733/) (2022)
- [An Ensemble Model with Attention Based Mechanism for Image Captioning](https://mlsanity.com/paper/2501.14828/) (2025)
- [Self-Supervised Image-to-Text and Text-to-Image Synthesis](https://mlsanity.com/paper/2112.04928/) (2021)
- [A Deep Neural Framework for Image Caption Generation Using GRU-Based Attention Mechanism](https://mlsanity.com/paper/2203.01594/) (2022)
- [How to Describe Images in a More Funny Way? Towards a Modular Approach to Cross-Modal Sarcasm Generation](https://mlsanity.com/paper/2211.10992/) (2022)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Pixels to Prose: Understanding the art of Image Captioning](https://mlsanity.com/paper/2408.15714/) (2024)
- [Attention-based transformer models for image captioning across languages: An in-depth survey and evaluation](https://mlsanity.com/paper/2506.05399/) (2025)
- [Image Captioning based on Feature Refinement and Reflective Decoding](https://mlsanity.com/paper/2206.07986/) (2022)
- [vieCap4H-VLSP 2021: Vietnamese Image Captioning for Healthcare Domain using Swin Transformer and Attention-based LSTM](https://mlsanity.com/paper/2209.01304/) (2022)
- [Stacked Cross-modal Feature Consolidation Attention Networks for Image Captioning](https://mlsanity.com/paper/2302.04676/) (2023)
- [A Novel Method of Fuzzy Topic Modeling based on Transformer Processing](https://mlsanity.com/paper/2309.09658/) (2023)
- [A Comparative Study of Pre-trained CNNs and GRU-Based Attention for Image Caption Generation](https://mlsanity.com/paper/2310.07252/) (2023)
- [A Wasserstein GAN-based climate scenario generator for risk management and insurance: the case of soil subsidence](https://mlsanity.com/paper/2605.06678/) (2026)

## Abstract (arXiv)

We propose "Areas of Attention", a novel attention-based model for automatic image captioning. Our approach models the dependencies between image regions, caption words, and the state of an RNN language model, using three pairwise interactions. In contrast to previous attention-based approaches that associate image regions only to the RNN state, our method allows a direct association between caption words and image regions. During training these associations are inferred from image-level captions, akin to weakly-supervised object detector training. These associations help to improve captioning by localizing the corresponding regions during testing. We also propose and compare different ways of generating attention areas: CNN activation grids, object proposals, and spatial transformers nets applied in a convolutional fashion. Spatial transformers give the best results. They allow for image specific attention areas, and can be trained jointly with the rest of the network. Our attention mechanism and spatial transformer attention areas together yield state-of-the-art results on the MSCOCO dataset.o meaningful latent semantic structure in the generated captions.