# TubeRMC: Tube-conditioned Reconstruction with Mutual Constraints for Weakly-supervised Spatio-Temporal Video Grounding

arXiv: https://arxiv.org/abs/2511.10241 · HTML: https://mlsanity.com/paper/2511.10241/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 13 Nov 2025.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences](https://mlsanity.com/paper/2001.06891/) (2020)
- [Human-centric Spatio-Temporal Video Grounding With Visual Transformers](https://mlsanity.com/paper/2011.05049/) (2020)
- [Weakly-Supervised Video Moment Retrieval via Semantic Completion Network](https://mlsanity.com/paper/1911.08199/) (2019)
- [Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video](https://mlsanity.com/paper/1906.02549/) (2019)
- [Context-Guided Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2401.01578/) (2024)
- [Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2209.13306/) (2022)
- [VLANet: Video-Language Alignment Network for Weakly-Supervised Video Moment Retrieval](https://mlsanity.com/paper/2008.10238/) (2020)
- [LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval](https://mlsanity.com/paper/1909.13784/) (2019)
- [Cross-Sentence Temporal and Semantic Relations in Video Activity Localisation](https://mlsanity.com/paper/2107.11443/) (2021)
- [Weakly Supervised Temporal Adjacent Network for Language Grounding](https://mlsanity.com/paper/2106.16136/) (2021)
- [Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2401.00901/) (2023)
- [Weakly-Supervised Multi-Level Attentional Reconstruction Network for Grounding Textual Queries in Videos](https://mlsanity.com/paper/2003.07048/) (2020)
- [ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations](https://mlsanity.com/paper/2501.14607/) (2025)
- [Object-Aware Multi-Branch Relation Networks for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2008.06941/) (2020)
- [Regularized Two-Branch Proposal Networks for Weakly-Supervised Moment Retrieval in Videos](https://mlsanity.com/paper/2008.08257/) (2020)
- [Augmented 2D-TAN: A Two-stage Approach for Human-centric Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2106.10634/) (2021)
- [STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding](https://mlsanity.com/paper/2502.20678/) (2025)
- [Gaussian Mixture Proposals with Pull-Push Learning Scheme to Capture Diverse Events for Weakly Supervised Temporal Video Grounding](https://mlsanity.com/paper/2312.16388/) (2023)
- [Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2510.10671/) (2025)

## Built on by (most cited first, 1 of 1)

- [Pocket-STVG: lightweight architecture for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2609.31135/) (2026)

## Abstract (arXiv)

Spatio-Temporal Video Grounding (STVG) aims to localize a spatio-temporal tube that corresponds to a given language query in an untrimmed video. This is a challenging task since it involves complex vision-language understanding and spatiotemporal reasoning. Recent works have explored weakly-supervised setting in STVG to eliminate reliance on fine-grained annotations like bounding boxes or temporal stamps. However, they typically follow a simple late-fusion manner, which generates tubes independent of the text description, often resulting in failed target identification and inconsistent target tracking. To address this limitation, we propose a Tube-conditioned Reconstruction with Mutual Constraints (TubeRMC) framework that generates text-conditioned candidate tubes with pre-trained visual grounding models and further refine them via tube-conditioned reconstruction with spatio-temporal constraints. Specifically, we design three reconstruction strategies from temporal, spatial, and spatio-temporal perspectives to comprehensively capture rich tube-text correspondences. Each strategy is equipped with a Tube-conditioned Reconstructor, utilizing spatio-temporal tubes as condition to reconstruct the key clues in the query. We further introduce mutual constraints between spatial and temporal proposals to enhance their quality for reconstruction. TubeRMC outperforms existing methods on two public benchmarks VidSTG and HCSTVG. Further visualization shows that TubeRMC effectively mitigates both target identification errors and inconsistent tracking.