# Referring to Objects in Videos using Spatio-Temporal Identifying Descriptions

arXiv: https://arxiv.org/abs/1904.03885 · HTML: https://mlsanity.com/paper/1904.03885/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2019.

## Built on by (most cited first, 1 of 1)

- [Decoupled Spatial Temporal Graphs for Generic Visual Grounding](https://mlsanity.com/paper/2103.10191/) (2021)

## Abstract (arXiv)

This paper presents a new task, the grounding of spatio-temporal identifying descriptions in videos. Previous work suggests potential bias in existing datasets and emphasizes the need for a new data creation schema to better model linguistic structure. We introduce a new data collection scheme based on grammatical constraints for surface realization to enable us to investigate the problem of grounding spatio-temporal identifying descriptions in videos. We then propose a two-stream modular attention network that learns and grounds spatio-temporal identifying descriptions based on appearance and motion. We show that motion modules help to ground motion-related words and also help to learn in appearance modules because modular neural networks resolve task interference between modules. Finally, we propose a future challenge and a need for a robust system arising from replacing ground truth visual annotations with automatic video object detector and temporal event localization.