# Attention Correctness in Neural Image Captioning

arXiv: https://arxiv.org/abs/1605.09553 · HTML: https://mlsanity.com/paper/1605.09553/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2016.

## Built on by (most cited first, 30 of 35)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [A General Survey on Attention Mechanisms in Deep Learning](https://mlsanity.com/paper/2203.14263/) (2022)
- [Measuring Representational Harms in Image Captioning](https://mlsanity.com/paper/2206.07173/) (2022)
- [Semantic Grouping Network for Video Captioning](https://mlsanity.com/paper/2102.00831/) (2021)
- [On Guiding Visual Attention with Language Specification](https://mlsanity.com/paper/2202.08926/) (2022)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering](https://mlsanity.com/paper/2110.01013/) (2021)
- [SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning](https://mlsanity.com/paper/2508.06125/) (2025)
- [Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images](https://mlsanity.com/paper/2110.00519/) (2021)
- [Prophet Attention: Predicting Attention with Future Attention for Image Captioning](https://mlsanity.com/paper/2210.10914/) (2022)
- [Keyword localisation in untranscribed speech using visually grounded speech models](https://mlsanity.com/paper/2202.01107/) (2022)
- [Guiding Visual Question Answering with Attention Priors](https://mlsanity.com/paper/2205.12616/) (2022)
- [Radiomics-Guided Global-Local Transformer for Weakly Supervised Pathology Localization in Chest X-Rays](https://mlsanity.com/paper/2207.04394/) (2022)
- [More Than Just Attention: Improving Cross-Modal Attentions with Contrastive Constraints for Image-Text Matching](https://mlsanity.com/paper/2105.09597/) (2021)
- [Guided Attention for Interpretable Motion Captioning](https://mlsanity.com/paper/2310.07324/) (2023)
- [Contrastive Language-Image Pretrained Models are Zero-Shot Human Scanpath Predictors](https://mlsanity.com/paper/2305.12380/) (2023)
- [Distributed Attention for Grounded Image Captioning](https://mlsanity.com/paper/2108.01056/) (2021)
- [CNN Attention Guidance for Improved Orthopedics Radiographic Fracture Classification](https://mlsanity.com/paper/2203.10690/) (2022)
- [Are metrics measuring what they should? An evaluation of image captioning task metrics](https://mlsanity.com/paper/2207.01733/) (2022)
- [Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval](https://mlsanity.com/paper/2607.04605/) (2026)
- [Transfer Reward Learning for Policy Gradient-Based Text Generation](https://mlsanity.com/paper/1909.03622/) (2019)
- [A Deep Neural Framework for Image Caption Generation Using GRU-Based Attention Mechanism](https://mlsanity.com/paper/2203.01594/) (2022)
- [Skeletal Human Action Recognition using Hybrid Attention based Graph Convolutional Network](https://mlsanity.com/paper/2207.05493/) (2022)
- [Language Cognition and Language Computation -- Human and Machine Language Understanding](https://mlsanity.com/paper/2301.04788/) (2023)
- [VisAlign: Dataset for Measuring the Degree of Alignment between AI and Humans in Visual Perception](https://mlsanity.com/paper/2308.01525/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [A Multi-Dataset Benchmark of Multiple Instance Learning for 3D Neuroimage Classification](https://mlsanity.com/paper/2604.26807/) (2026)
- [From Coarse to Fine-grained Concept based Discrimination for Phrase Detection](https://mlsanity.com/paper/2112.03237/) (2021)
- [Synthesizer Based Efficient Self-Attention for Vision Tasks](https://mlsanity.com/paper/2201.01410/) (2022)
- [An Image captioning algorithm based on the Hybrid Deep Learning Technique (CNN+GRU)](https://mlsanity.com/paper/2301.02440/) (2023)

## Abstract (arXiv)

Attention mechanisms have recently been introduced in deep learning for various tasks in natural language processing and computer vision. But despite their popularity, the "correctness" of the implicitly-learned attention maps has only been assessed qualitatively by visualization of several examples. In this paper we focus on evaluating and improving the correctness of attention in neural image captioning models. Specifically, we propose a quantitative evaluation metric for the consistency between the generated attention maps and human annotations, using recently released datasets with alignment between regions in images and entities in captions. We then propose novel models with different levels of explicit supervision for learning attention maps during training. The supervision can be strong when alignment between regions and caption entities are available, or weak when only object segments and categories are provided. We show on the popular Flickr30k and COCO datasets that introducing supervision of attention maps during training solidly improves both attention correctness and caption quality, showing the promise of making machine perception more human-like.