# Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding

arXiv: https://arxiv.org/abs/2401.00901 · HTML: https://mlsanity.com/paper/2401.00901/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 31 Dec 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Language Is Not All You Need: Aligning Perception with Language Models](https://mlsanity.com/paper/2302.14045/) (2023)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences](https://mlsanity.com/paper/2001.06891/) (2020)
- [Grounding of Textual Phrases in Images by Reconstruction](https://mlsanity.com/paper/1511.03745/) (2015)
- [TubeDETR: Spatio-Temporal Video Grounding with Transformers](https://mlsanity.com/paper/2203.16434/) (2022)
- [Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding](https://mlsanity.com/paper/2109.04872/) (2021)
- [Human-centric Spatio-Temporal Video Grounding With Visual Transformers](https://mlsanity.com/paper/2011.05049/) (2020)
- [Proposal-free Temporal Moment Localization of a Natural-Language Query in Video using Guided Attention](https://mlsanity.com/paper/1908.07236/) (2019)
- [Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video](https://mlsanity.com/paper/1906.02549/) (2019)
- [Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2209.13306/) (2022)
- [Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction](https://mlsanity.com/paper/1805.02834/) (2018)
- [Spatio-temporal Person Retrieval via Natural Language Queries](https://mlsanity.com/paper/1704.07945/) (2017)
- [Object-Aware Multi-Branch Relation Networks for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2008.06941/) (2020)
- [Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos](https://mlsanity.com/paper/2110.10596/) (2021)
- [Augmented 2D-TAN: A Two-stage Approach for Human-centric Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2106.10634/) (2021)
- [What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions](https://mlsanity.com/paper/2303.16990/) (2023)
- [2rd Place Solutions in the HC-STVG track of Person in Context Challenge 2021](https://mlsanity.com/paper/2106.07166/) (2021)

## Built on by (most cited first, 30 of 32)

- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Locate Anything on Earth: Advancing Open-Vocabulary Object Detection for Remote Sensing Community](https://mlsanity.com/paper/2408.09110/) (2024)
- [ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations](https://mlsanity.com/paper/2501.14607/) (2025)
- [VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos](https://mlsanity.com/paper/2510.19488/) (2025)
- [Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2502.11168/) (2025)
- [TimeRefine: Temporal Grounding with Time Refining Video LLM](https://mlsanity.com/paper/2412.09601/) (2024)
- [Anomize: Better Open Vocabulary Video Anomaly Detection](https://mlsanity.com/paper/2503.18094/) (2025)
- [Sherlock: Towards Multi-scene Video Abnormal Event Extraction and Localization via a Global-local Spatial-sensitive LLM](https://mlsanity.com/paper/2502.18863/) (2025)
- [STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding](https://mlsanity.com/paper/2502.20678/) (2025)
- [Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2501.17053/) (2025)
- [Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2509.15178/) (2025)
- [ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videos](https://mlsanity.com/paper/2512.03666/) (2025)
- [OmniSTVG: Toward Spatio-Temporal Omni-Object Video Grounding](https://mlsanity.com/paper/2503.10500/) (2025)
- [Large-scale Pre-training for Grounded Video Caption Generation](https://mlsanity.com/paper/2503.10781/) (2025)
- [STING-BEE: Towards Vision-Language Model for Real-World X-ray Baggage Security Inspection](https://mlsanity.com/paper/2504.02823/) (2025)
- [DeCafNet: Delegate and Conquer for Efficient Temporal Grounding in Long Videos](https://mlsanity.com/paper/2505.16376/) (2025)
- [Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding](https://mlsanity.com/paper/2604.08014/) (2026)
- [Towards Long-Form Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2602.23294/) (2026)
- [TubeRMC: Tube-conditioned Reconstruction with Mutual Constraints for Weakly-supervised Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2511.10241/) (2025)
- [Can a Second-View Image Be a Language? Geometric and Semantic Cross-Modal Reasoning for X-ray Prohibited Item Detection](https://mlsanity.com/paper/2511.18385/) (2025)
- [Detector-Empowered Video Large Language Model for Efficient Spatio-Temporal Grounding](https://mlsanity.com/paper/2512.06673/) (2025)
- [Grounded Video Caption Generation](https://mlsanity.com/paper/2411.07584/) (2024)
- [VideoGEM: Training-free Action Grounding in Videos](https://mlsanity.com/paper/2503.20348/) (2025)
- [CGTGait: Collaborative Graph and Transformer for Gait Emotion Recognition](https://mlsanity.com/paper/2509.16623/) (2025)
- [Unlocking the Potential of Grounding DINO in Videos: Parameter-Efficient Adaptation for Limited-Data Spatial-Temporal Localization](https://mlsanity.com/paper/2604.12346/) (2026)
- [Training-Free Open-Vocabulary Visual Grounding for Remote Sensing Images and Videos](https://mlsanity.com/paper/2606.16124/) (2026)
- [Toward Low-Latency Vision-Language Models with Doubly-Correct Predictions in Egocentric Visual Understanding](https://mlsanity.com/paper/2606.25160/) (2026)
- [Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification](https://mlsanity.com/paper/2606.29023/) (2026)
- [AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models](https://mlsanity.com/paper/2607.02269/) (2026)
- [ScanFocus: A Coarse-to-Fine Framework for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2607.13421/) (2026)

## Abstract (arXiv)

Video grounding aims to localize a spatio-temporal section in a video corresponding to an input text query. This paper addresses a critical limitation in current video grounding methodologies by introducing an Open-Vocabulary Spatio-Temporal Video Grounding task. Unlike prevalent closed-set approaches that struggle with open-vocabulary scenarios due to limited training data and predefined vocabularies, our model leverages pre-trained representations from foundational spatial grounding models. This empowers it to effectively bridge the semantic gap between natural language and diverse visual content, achieving strong performance in closed-set and open-vocabulary settings. Our contributions include a novel spatio-temporal video grounding model, surpassing state-of-the-art results in closed-set evaluations on multiple datasets and demonstrating superior performance in open-vocabulary scenarios. Notably, the proposed model outperforms state-of-the-art methods in closed-set settings on VidSTG (Declarative and Interrogative) and HC-STVG (V1 and V2) datasets. Furthermore, in open-vocabulary evaluations on HC-STVG V1 and YouCook-Interactions, our model surpasses the recent best-performing models by 4.88 mvIoU and 1.83% accuracy, demonstrating its efficacy in handling diverse linguistic and visual concepts for improved video understanding. Our codes will be publicly released.