# Described Spatial-Temporal Video Detection

arXiv: https://arxiv.org/abs/2407.05610 · HTML: https://mlsanity.com/paper/2407.05610/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [Improving One-stage Visual Grounding by Recursive Sub-query Construction](https://mlsanity.com/paper/2008.01059/) (2020)
- [Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/1910.14303/) (2019)
- [Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences](https://mlsanity.com/paper/2001.06891/) (2020)
- [TubeDETR: Spatio-Temporal Video Grounding with Transformers](https://mlsanity.com/paper/2203.16434/) (2022)
- [Human-centric Spatio-Temporal Video Grounding With Visual Transformers](https://mlsanity.com/paper/2011.05049/) (2020)
- [UnLoc: A Unified Framework for Video Localization Tasks](https://mlsanity.com/paper/2308.11062/) (2023)
- [Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video](https://mlsanity.com/paper/1906.02549/) (2019)
- [Finding Action Tubes](https://mlsanity.com/paper/1411.6031/) (2014)
- [Unified Coarse-to-Fine Alignment for Video-Text Retrieval](https://mlsanity.com/paper/2309.10091/) (2023)
- [A Content-Driven Micro-Video Recommendation Dataset at Scale](https://mlsanity.com/paper/2309.15379/) (2023)
- [Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2209.13306/) (2022)
- [Spatio-temporal Person Retrieval via Natural Language Queries](https://mlsanity.com/paper/1704.07945/) (2017)
- [Video Object Grounding using Semantic Roles in Language Description](https://mlsanity.com/paper/2003.10606/) (2020)
- [Video Action Detection: Analysing Limitations and Challenges](https://mlsanity.com/paper/2204.07892/) (2022)
- [What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions](https://mlsanity.com/paper/2303.16990/) (2023)

## Built on by (most cited first, 3 of 3)

- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videos](https://mlsanity.com/paper/2512.03666/) (2025)
- [OmniSTVG: Toward Spatio-Temporal Omni-Object Video Grounding](https://mlsanity.com/paper/2503.10500/) (2025)

## Abstract (arXiv)

Detecting visual content on language expression has become an emerging topic in the community. However, in the video domain, the existing setting, i.e., spatial-temporal video grounding (STVG), is formulated to only detect one pre-existing object in each frame, ignoring the fact that language descriptions can involve none or multiple entities within a video. In this work, we advance the STVG to a more practical setting called described spatial-temporal video detection (DSTVD) by overcoming the above limitation. To facilitate the exploration of DSTVD, we first introduce a new benchmark, namely DVD-ST. Notably, DVD-ST supports grounding from none to many objects onto the video in response to queries and encompasses a diverse range of over 150 entities, including appearance, actions, locations, and interactions. The extensive breadth and diversity of the DVD-ST dataset make it an exemplary testbed for the investigation of DSTVD. In addition to the new benchmark, we further present two baseline methods for our proposed DSTVD task by extending two representative STVG models, i.e., TubeDETR, and STCAT. These extended models capitalize on tubelet queries to localize and track referred objects across the video sequence. Besides, we adjust the training objectives of these models to optimize spatial and temporal localization accuracy and multi-class classification capabilities. Furthermore, we benchmark the baselines on the introduced DVD-ST dataset and conduct extensive experimental analysis to guide future investigation. Our code and benchmark will be publicly available.