# Spatio-Temporal Attention Models for Grounded Video Captioning

arXiv: https://arxiv.org/abs/1610.04997 · HTML: https://mlsanity.com/paper/1610.04997/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2016.

## Built on by (most cited first, 8 of 8)

- [Attention Mechanism in Neural Networks: Where it Comes and Where it Goes](https://mlsanity.com/paper/2204.13154/) (2022)
- [Consensus Graph Representation Learning for Better Grounded Image Captioning](https://mlsanity.com/paper/2112.00974/) (2021)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Towards Diverse Paragraph Captioning for Untrimmed Videos](https://mlsanity.com/paper/2105.14477/) (2021)
- [Relational Graph Learning for Grounded Video Description Generation](https://mlsanity.com/paper/2112.00967/) (2021)
- [Large-scale Pre-training for Grounded Video Caption Generation](https://mlsanity.com/paper/2503.10781/) (2025)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [Co-segmentation Inspired Attention Module for Video-based Computer Vision Tasks](https://mlsanity.com/paper/2111.07370/) (2021)

## Abstract (arXiv)

Automatic video captioning is challenging due to the complex interactions in dynamic real scenes. A comprehensive system would ultimately localize and track the objects, actions and interactions present in a video and generate a description that relies on temporal localization in order to ground the visual concepts. However, most existing automatic video captioning systems map from raw video data to high level textual description, bypassing localization and recognition, thus discarding potentially valuable information for content localization and generalization. In this work we present an automatic video captioning model that combines spatio-temporal attention and image classification by means of deep neural network structures based on long short-term memory. The resulting system is demonstrated to produce state-of-the-art results in the standard YouTube captioning benchmark while also offering the advantage of localizing the visual concepts (subjects, verbs, objects), with no grounding supervision, over space and time.