# Text-guided Attention Model for Image Captioning

arXiv: https://arxiv.org/abs/1612.03557 · HTML: https://mlsanity.com/paper/1612.03557/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Dec 2016.

## Built on by (most cited first, 13 of 13)

- [Visual Navigation with Spatial Attention](https://mlsanity.com/paper/2104.09807/) (2021)
- [Noise-aware Learning from Web-crawled Image-Text Data for Image Captioning](https://mlsanity.com/paper/2212.13563/) (2022)
- [See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization](https://mlsanity.com/paper/2105.09601/) (2021)
- [Trajectory Prediction with Linguistic Representations](https://mlsanity.com/paper/2110.09741/) (2021)
- [HICEScore: A Hierarchical Metric for Image Captioning Evaluation](https://mlsanity.com/paper/2407.18589/) (2024)
- [Neural Attention for Image Captioning: Review of Outstanding Methods](https://mlsanity.com/paper/2111.15015/) (2021)
- [SALAD: Source-free Active Label-Agnostic Domain Adaptation for Classification, Segmentation and Detection](https://mlsanity.com/paper/2205.12840/) (2022)
- [Fusing Multimodal Signals on Hyper-complex Space for Extreme Abstractive Text Summarization (TL;DR) of Scientific Contents](https://mlsanity.com/paper/2306.13968/) (2023)
- [Contrastive Language-Image Pretrained Models are Zero-Shot Human Scanpath Predictors](https://mlsanity.com/paper/2305.12380/) (2023)
- [LuoJiaHOG: A Hierarchy Oriented Geo-aware Image Caption Dataset for Remote Sensing Image-Text Retrival](https://mlsanity.com/paper/2403.10887/) (2024)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [EventFormer: A Node-graph Hierarchical Attention Transformer for Action-centric Video Event Prediction](https://mlsanity.com/paper/2510.21786/) (2025)
- [Observing Health Outcomes Using Remote Sensing Imagery and Geo-Context Guided Visual Transformer](https://mlsanity.com/paper/2602.00110/) (2026)

## Abstract (arXiv)

Visual attention plays an important role to understand images and demonstrates its effectiveness in generating natural language descriptions of images. On the other hand, recent studies show that language associated with an image can steer visual attention in the scene during our cognitive process. Inspired by this, we introduce a text-guided attention model for image captioning, which learns to drive visual attention using associated captions. For this model, we propose an exemplar-based learning approach that retrieves from training data associated captions with each image, and use them to learn attention on visual features. Our attention model enables to describe a detailed state of scenes by distinguishing small or confusable objects effectively. We validate our model on MS-COCO Captioning benchmark and achieve the state-of-the-art performance in standard metrics.