# Stack-VS: Stacked Visual-Semantic Attention for Image Caption Generation

arXiv: https://arxiv.org/abs/1909.02489 · HTML: https://mlsanity.com/paper/1909.02489/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2019.

## Built on by (most cited first, 7 of 7)

- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception](https://mlsanity.com/paper/2504.06666/) (2025)
- [CLIP-Diffusion-LM: Apply Diffusion Model on Image Captioning](https://mlsanity.com/paper/2210.04559/) (2022)
- [Declaration-based Prompt Tuning for Visual Question Answering](https://mlsanity.com/paper/2205.02456/) (2022)
- [DiffCap: Exploring Continuous Diffusion on Image Captioning](https://mlsanity.com/paper/2305.12144/) (2023)
- [Caption Generation on Scenes with Seen and Unseen Object Categories](https://mlsanity.com/paper/2108.06165/) (2021)

## Abstract (arXiv)

Recently, automatic image caption generation has been an important focus of the work on multimodal translation task. Existing approaches can be roughly categorized into two classes, i.e., top-down and bottom-up, the former transfers the image information (called as visual-level feature) directly into a caption, and the later uses the extracted words (called as semanticlevel attribute) to generate a description. However, previous methods either are typically based one-stage decoder or partially utilize part of visual-level or semantic-level information for image caption generation. In this paper, we address the problem and propose an innovative multi-stage architecture (called as Stack-VS) for rich fine-gained image caption generation, via combining bottom-up and top-down attention models to effectively handle both visual-level and semantic-level information of an input image. Specifically, we also propose a novel well-designed stack decoder model, which is constituted by a sequence of decoder cells, each of which contains two LSTM-layers work interactively to re-optimize attention weights on both visual-level feature vectors and semantic-level attribute embeddings for generating a fine-gained image caption. Extensive experiments on the popular benchmark dataset MSCOCO show the significant improvements on different evaluation metrics, i.e., the improvements on BLEU-4/CIDEr/SPICE scores are 0.372, 1.226 and 0.216, respectively, as compared to the state-of-the-arts.