# ChatVTG: Video Temporal Grounding via Chat with Video Dialogue Large Language Models

arXiv: https://arxiv.org/abs/2410.12813 · HTML: https://mlsanity.com/paper/2410.12813/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [VTimeLLM: Empower LLM to Grasp Video Moments](https://mlsanity.com/paper/2311.18445/) (2023)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023)
- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding](https://mlsanity.com/paper/2109.04872/) (2021)
- [Weakly Supervised Video Moment Retrieval From Text Queries](https://mlsanity.com/paper/1904.03282/) (2019)
- [CTVIS: Consistent Training for Online Video Instance Segmentation](https://mlsanity.com/paper/2307.12616/) (2023)
- [PG-Video-LLaVA: Pixel Grounding Large Video-Language Models](https://mlsanity.com/paper/2311.13435/) (2023)
- [Cross-Sentence Temporal and Semantic Relations in Video Activity Localisation](https://mlsanity.com/paper/2107.11443/) (2021)
- [Zero-shot Natural Language Video Localization](https://mlsanity.com/paper/2110.00428/) (2021)
- [Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Training](https://mlsanity.com/paper/2303.00040/) (2023)
- [Unsupervised Temporal Video Grounding with Deep Semantic Clustering](https://mlsanity.com/paper/2201.05307/) (2022)
- [Language-free Training for Zero-shot Video Grounding](https://mlsanity.com/paper/2210.12977/) (2022)
- [LLM4VG: Large Language Models Evaluation for Video Grounding](https://mlsanity.com/paper/2312.14206/) (2023)
- [Part-level Action Parsing via a Pose-guided Coarse-to-Fine Framework](https://mlsanity.com/paper/2203.04476/) (2022)

## Built on by (most cited first, 30 of 54)

- [InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling](https://mlsanity.com/paper/2501.12386/) (2025)
- [Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding](https://mlsanity.com/paper/2503.13377/) (2025)
- [From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D](https://mlsanity.com/paper/2503.22976/) (2025)
- [TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning](https://mlsanity.com/paper/2410.19702/) (2024)
- [TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability](https://mlsanity.com/paper/2411.18211/) (2024)
- [VideoWorld: Exploring Knowledge Learning from Unlabeled Videos](https://mlsanity.com/paper/2501.09781/) (2025)
- [VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning](https://mlsanity.com/paper/2503.13444/) (2025)
- [MotionPro: A Precise Motion Controller for Image-to-Video Generation](https://mlsanity.com/paper/2505.20287/) (2025)
- [Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment](https://mlsanity.com/paper/2412.19326/) (2024)
- [LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs](https://mlsanity.com/paper/2603.19217/) (2026)
- [A Survey on Video Temporal Grounding with Multimodal Large Language Model](https://mlsanity.com/paper/2508.10922/) (2025)
- [ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos](https://mlsanity.com/paper/2411.14901/) (2024)
- [Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2501.04670/) (2025)
- [AURA: Always-On Understanding and Real-Time Assistance via Video Streams](https://mlsanity.com/paper/2604.04184/) (2026)
- [TimeRefine: Temporal Grounding with Time Refining Video LLM](https://mlsanity.com/paper/2412.09601/) (2024)
- [TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action](https://mlsanity.com/paper/2505.01583/) (2025)
- [Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation](https://mlsanity.com/paper/2504.01020/) (2025)
- [Active Perception Agent for Omnimodal Audio-Video Understanding](https://mlsanity.com/paper/2512.23646/) (2025)
- [Enrich and Detect: Video Temporal Grounding with Multimodal LLMs](https://mlsanity.com/paper/2510.17023/) (2025)
- [Object-Shot Enhanced Grounding Network for Egocentric Video](https://mlsanity.com/paper/2505.04270/) (2025)
- [TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2512.03963/) (2025)
- [DisTime: Distribution-based Time Representation for Video Large Language Models](https://mlsanity.com/paper/2505.24329/) (2025)
- [VideoTG-R1: Boosting Video Temporal Grounding via Curriculum Reinforcement Learning on Reflected Boundary Annotations](https://mlsanity.com/paper/2510.23397/) (2025)
- [VideoTemp-o3: Harmonizing Temporal Grounding and Video Understanding in Agentic Thinking-with-Videos](https://mlsanity.com/paper/2602.07801/) (2026)
- [MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answering](https://mlsanity.com/paper/2506.18071/) (2025)
- [VeRVE: Versatile Retrieval for Videos via Unified Embeddings](https://mlsanity.com/paper/2601.12193/) (2026)
- [OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding](https://mlsanity.com/paper/2604.25276/) (2026)
- [Harnessing Streaming Video in the Wild](https://mlsanity.com/paper/2606.08615/) (2026)
- [Video-in-the-Loop: Span-Grounded Long Video QA with Interleaved Reasoning](https://mlsanity.com/paper/2510.04022/) (2025)
- [SVAG-Bench: A Large-Scale Benchmark for Multi-Instance Spatio-temporal Video Action Grounding](https://mlsanity.com/paper/2510.13016/) (2025)

## Abstract (arXiv)

Video Temporal Grounding (VTG) aims to ground specific segments within an untrimmed video corresponding to the given natural language query. Existing VTG methods largely depend on supervised learning and extensive annotated data, which is labor-intensive and prone to human biases. To address these challenges, we present ChatVTG, a novel approach that utilizes Video Dialogue Large Language Models (LLMs) for zero-shot video temporal grounding. Our ChatVTG leverages Video Dialogue LLMs to generate multi-granularity segment captions and matches these captions with the given query for coarse temporal grounding, circumventing the need for paired annotation data. Furthermore, to obtain more precise temporal grounding results, we employ moment refinement for fine-grained caption proposals. Extensive experiments on three mainstream VTG datasets, including Charades-STA, ActivityNet-Captions, and TACoS, demonstrate the effectiveness of ChatVTG. Our ChatVTG surpasses the performance of current zero-shot methods.