# Adaptively Aligned Image Captioning via Adaptive Attention Time

arXiv: https://arxiv.org/abs/1909.09060 · HTML: https://mlsanity.com/paper/1909.09060/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2019.

## Built on by (most cited first, 15 of 15)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [MeaCap: Memory-Augmented Zero-shot Image Captioning](https://mlsanity.com/paper/2403.03715/) (2024)
- [ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing](https://mlsanity.com/paper/2303.02437/) (2023)
- [Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning](https://mlsanity.com/paper/2205.04363/) (2022)
- [Exploiting Cross-Modal Prediction and Relation Consistency for Semi-Supervised Image Captioning](https://mlsanity.com/paper/2110.11767/) (2021)
- [VIXEN: Visual Text Comparison Network for Image Difference Captioning](https://mlsanity.com/paper/2402.19119/) (2024)
- [Neural Attention for Image Captioning: Review of Outstanding Methods](https://mlsanity.com/paper/2111.15015/) (2021)
- [Prophet Attention: Predicting Attention with Future Attention for Image Captioning](https://mlsanity.com/paper/2210.10914/) (2022)
- [A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes](https://mlsanity.com/paper/2403.07469/) (2024)
- [Understanding Social Media Cross-Modality Discourse in Linguistic Space](https://mlsanity.com/paper/2302.13311/) (2023)
- [A Comprehensive Analysis of Real-World Image Captioning and Scene Identification](https://mlsanity.com/paper/2308.02833/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Models](https://mlsanity.com/paper/2505.24227/) (2025)
- [Mirror-Yolo: A Novel Attention Focus, Instance Segmentation and Mirror Detection Model](https://mlsanity.com/paper/2202.08498/) (2022)
- [Semi-Supervised Image Captioning Considering Wasserstein Graph Matching](https://mlsanity.com/paper/2403.17995/) (2024)

## Abstract (arXiv)

Recent neural models for image captioning usually employ an encoder-decoder framework with an attention mechanism. However, the attention mechanism in such a framework aligns one single (attended) image feature vector to one caption word, assuming one-to-one mapping from source image regions and target caption words, which is never possible. In this paper, we propose a novel attention model, namely Adaptive Attention Time (AAT), to align the source and the target adaptively for image captioning. AAT allows the framework to learn how many attention steps to take to output a caption word at each decoding step. With AAT, an image region can be mapped to an arbitrary number of caption words while a caption word can also attend to an arbitrary number of image regions. AAT is deterministic and differentiable, and doesn't introduce any noise to the parameter gradients. In this paper, we empirically show that AAT improves over state-of-the-art methods on the task of image captioning. Code is available at https://github.com/husthuaan/AAT.