# Spatio-Temporal Ranked-Attention Networks for Video Captioning

arXiv: https://arxiv.org/abs/2001.06127 · HTML: https://mlsanity.com/paper/2001.06127/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2020.

## Built on by (most cited first, 6 of 6)

- [(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering](https://mlsanity.com/paper/2202.09277/) (2022)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Multi-modal Mood Reader: Pre-trained Model Empowers Cross-Subject Emotion Recognition](https://mlsanity.com/paper/2405.19373/) (2024)
- [Diverse Video Captioning by Adaptive Spatio-temporal Attention](https://mlsanity.com/paper/2208.09266/) (2022)
- [Visual-aware Attention Dual-stream Decoder for Video Captioning](https://mlsanity.com/paper/2110.08578/) (2021)
- [Flexible and Efficient Spatio-Temporal Transformer for Sequential Visual Place Recognition](https://mlsanity.com/paper/2510.04282/) (2025)

## Abstract (arXiv)

Generating video descriptions automatically is a challenging task that involves a complex interplay between spatio-temporal visual features and language models. Given that videos consist of spatial (frame-level) features and their temporal evolutions, an effective captioning model should be able to attend to these different cues selectively. To this end, we propose a Spatio-Temporal and Temporo-Spatial (STaTS) attention model which, conditioned on the language state, hierarchically combines spatial and temporal attention to videos in two different orders: (i) a spatio-temporal (ST) sub-model, which first attends to regions that have temporal evolution, then temporally pools the features from these regions; and (ii) a temporo-spatial (TS) sub-model, which first decides a single frame to attend to, then applies spatial attention within that frame. We propose a novel LSTM-based temporal ranking function, which we call ranked attention, for the ST model to capture action dynamics. Our entire framework is trained end-to-end. We provide experiments on two benchmark datasets: MSVD and MSR-VTT. Our results demonstrate the synergy between the ST and TS modules, outperforming recent state-of-the-art methods.