# Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Grounding

arXiv: https://arxiv.org/abs/2502.11168 · HTML: https://mlsanity.com/paper/2502.11168/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 55 references that are in the map.
- First posted on arXiv 16 Feb 2025.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [GRES: Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2306.00968/) (2023)
- [PixelLM: Pixel Reasoning with Large Multimodal Model](https://mlsanity.com/paper/2312.02228/) (2023)
- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [Local-Global Video-Text Interactions for Temporal Grounding](https://mlsanity.com/paper/2004.07514/) (2020)
- [Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks](https://mlsanity.com/paper/1911.07883/) (2019)
- [Rethinking Transformer-based Set Prediction for Object Detection](https://mlsanity.com/paper/2011.10881/) (2020)
- [In Defense of Grid Features for Visual Question Answering](https://mlsanity.com/paper/2001.03615/) (2020)
- [LongVLM: Efficient Long Video Understanding via Large Language Models](https://mlsanity.com/paper/2404.03384/) (2024)
- [Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences](https://mlsanity.com/paper/2001.06891/) (2020)
- [TubeDETR: Spatio-Temporal Video Grounding with Transformers](https://mlsanity.com/paper/2203.16434/) (2022)
- [Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding](https://mlsanity.com/paper/2109.04872/) (2021)
- [Human-centric Spatio-Temporal Video Grounding With Visual Transformers](https://mlsanity.com/paper/2011.05049/) (2020)
- [Joint Visual Grounding and Tracking with Natural Language Specification](https://mlsanity.com/paper/2303.12027/) (2023)
- [Hard Patches Mining for Masked Image Modeling](https://mlsanity.com/paper/2304.05919/) (2023)
- [Reconstructive Visual Instruction Tuning](https://mlsanity.com/paper/2410.09575/) (2024)
- [Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos](https://mlsanity.com/paper/2312.10300/) (2023)
- [End-to-end Multi-modal Video Temporal Grounding](https://mlsanity.com/paper/2107.05624/) (2021)
- [Divert More Attention to Vision-Language Tracking](https://mlsanity.com/paper/2207.01076/) (2022)

## Built on by (most cited first, 18 of 18)

- [VGR: Visual Grounded Reasoning](https://mlsanity.com/paper/2506.11991/) (2025)
- [Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2511.21375/) (2025)
- [ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videos](https://mlsanity.com/paper/2512.03666/) (2025)
- [OmniSTVG: Toward Spatio-Temporal Omni-Object Video Grounding](https://mlsanity.com/paper/2503.10500/) (2025)
- [OmniGround: A Comprehensive Spatio-Temporal Grounding Benchmark for Real-World Complex Scenarios](https://mlsanity.com/paper/2511.16937/) (2025)
- [STVG-R1: Incentivizing Instance-Level Reasoning and Grounding in Videos via Reinforcement Learning](https://mlsanity.com/paper/2602.11730/) (2026)
- [Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2510.10671/) (2025)
- [Vidi2.5: Large Multimodal Models for Video Understanding and Creation](https://mlsanity.com/paper/2511.19529/) (2025)
- [Towards Long-Form Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2602.23294/) (2026)
- [Detector-Empowered Video Large Language Model for Efficient Spatio-Temporal Grounding](https://mlsanity.com/paper/2512.06673/) (2025)
- [Knowledge-Preserved Model Tuning in Null-Space for Robust Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2606.03539/) (2026)
- [Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification](https://mlsanity.com/paper/2606.29023/) (2026)
- [AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models](https://mlsanity.com/paper/2607.02269/) (2026)
- [ScanFocus: A Coarse-to-Fine Framework for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2607.13421/) (2026)
- [Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2608.28192/) (2026)
- [Learning Compositional Spatio-Temporal Video Grounding with Synthetic Curriculum](https://mlsanity.com/paper/2608.30584/) (2026)
- [Pocket-STVG: lightweight architecture for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2609.31135/) (2026)
- [Investigating Query-Insensitive Behavior in Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2610.06018/) (2026)

## Abstract (arXiv)

Transformer has attracted increasing interest in STVG, owing to its end-to-end pipeline and promising result. Existing Transformer-based STVG approaches often leverage a set of object queries, which are initialized simply using zeros and then gradually learn target position information via iterative interactions with multimodal features, for spatial and temporal localization. Despite simplicity, these zero object queries, due to lacking target-specific cues, are hard to learn discriminative target information from interactions with multimodal features in complicated scenarios (e.g., with distractors or occlusion), resulting in degradation. Addressing this, we introduce a novel Target-Aware Transformer for STVG (TA-STVG), which seeks to adaptively generate object queries via exploring target-specific cues from the given video-text pair, for improving STVG. The key lies in two simple yet effective modules, comprising text-guided temporal sampling (TTS) and attribute-aware spatial activation (ASA), working in a cascade. The former focuses on selecting target-relevant temporal cues from a video utilizing holistic text information, while the latter aims at further exploiting the fine-grained visual attribute information of the object from previous target-aware temporal cues, which is applied for object query initialization. Compared to existing methods leveraging zero-initialized queries, object queries in our TA-STVG, directly generated from a given video-text pair, naturally carry target-specific cues, making them adaptive and better interact with multimodal features for learning more discriminative information to improve STVG. In our experiments on three benchmarks, TA-STVG achieves state-of-the-art performance and significantly outperforms the baseline, validating its efficacy.