# Spatio-temporal Person Retrieval via Natural Language Queries

arXiv: https://arxiv.org/abs/1704.07945 · HTML: https://mlsanity.com/paper/1704.07945/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2017.

## Built on by (most cited first, 25 of 25)

- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [TubeDETR: Spatio-Temporal Video Grounding with Transformers](https://mlsanity.com/paper/2203.16434/) (2022)
- [Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2209.13306/) (2022)
- [Collaborative Spatial-Temporal Modeling for Language-Queried Video Actor Segmentation](https://mlsanity.com/paper/2105.06818/) (2021)
- [Weakly-Supervised Spatio-Temporal Anomaly Detection in Surveillance Video](https://mlsanity.com/paper/2108.03825/) (2021)
- [Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2401.00901/) (2023)
- [Referring Segmentation in Images and Videos with Cross-Modal Self-Attention Network](https://mlsanity.com/paper/2102.04762/) (2021)
- [RefEgo: Referring Expression Comprehension Dataset from First-Person Perception of Ego4D](https://mlsanity.com/paper/2308.12035/) (2023)
- [End-to-End Modeling via Information Tree for One-Shot Natural Language Spatial Video Grounding](https://mlsanity.com/paper/2203.08013/) (2022)
- [Connecting Language and Vision for Natural Language-Based Vehicle Retrieval](https://mlsanity.com/paper/2105.14897/) (2021)
- [4-LEGS: 4D Language Embedded Gaussian Splatting](https://mlsanity.com/paper/2410.10719/) (2024)
- [STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding](https://mlsanity.com/paper/2502.20678/) (2025)
- [Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2501.17053/) (2025)
- [ActionVOS: Actions as Prompts for Video Object Segmentation](https://mlsanity.com/paper/2407.07402/) (2024)
- [ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videos](https://mlsanity.com/paper/2512.03666/) (2025)
- [OmniSTVG: Toward Spatio-Temporal Omni-Object Video Grounding](https://mlsanity.com/paper/2503.10500/) (2025)
- [Person Retrieval in Surveillance Using Textual Query: A Review](https://mlsanity.com/paper/2105.02414/) (2021)
- [Understanding Synonymous Referring Expressions via Contrastive Features](https://mlsanity.com/paper/2104.10156/) (2021)
- [SBNet: Segmentation-based Network for Natural Language-based Vehicle Search](https://mlsanity.com/paper/2104.11589/) (2021)
- [Gaussian Kernel-based Cross Modal Network for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2207.00744/) (2022)
- [Described Spatial-Temporal Video Detection](https://mlsanity.com/paper/2407.05610/) (2024)
- [Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification](https://mlsanity.com/paper/2606.29023/) (2026)
- [AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models](https://mlsanity.com/paper/2607.02269/) (2026)
- [Pocket-STVG: lightweight architecture for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2609.31135/) (2026)
- [Investigating Query-Insensitive Behavior in Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2610.06018/) (2026)

## Abstract (arXiv)

In this paper, we address the problem of spatio-temporal person retrieval from multiple videos using a natural language query, in which we output a tube (i.e., a sequence of bounding boxes) which encloses the person described by the query. For this problem, we introduce a novel dataset consisting of videos containing people annotated with bounding boxes for each second and with five natural language descriptions. To retrieve the tube of the person described by a given natural language query, we design a model that combines methods for spatio-temporal human detection and multimodal retrieval. We conduct comprehensive experiments to compare a variety of tube and text representations and multimodal retrieval methods, and present a strong baseline in this task as well as demonstrate the efficacy of our tube representation and multimodal feature embedding technique. Finally, we demonstrate the versatility of our model by applying it to two other important tasks.