# UniVTG: Towards Unified Video-Language Temporal Grounding

arXiv: https://arxiv.org/abs/2307.16715 · HTML: https://mlsanity.com/paper/2307.16715/

## Key facts

- Cited by 190 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 70 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 30 of 45)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [GLIPv2: Unifying Localization and Vision-Language Understanding](https://mlsanity.com/paper/2206.05836/) (2022)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2206.01670/) (2022)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [Learning Video Representations from Large Language Models](https://mlsanity.com/paper/2212.04501/) (2022)
- [Local-Global Video-Text Interactions for Temporal Grounding](https://mlsanity.com/paper/2004.07514/) (2020)
- [Dense Regression Network for Video Grounding](https://mlsanity.com/paper/2004.03545/) (2020)
- [All in One: Exploring Unified Video-Language Pre-training](https://mlsanity.com/paper/2203.07303/) (2022)
- [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2303.13874/) (2023)
- [UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2203.12745/) (2022)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone](https://mlsanity.com/paper/2307.05463/) (2023)
- [ExCL: Extractive Clip Localization Using Natural Language Descriptions](https://mlsanity.com/paper/1904.02755/) (2019)
- [Learning Audio-Video Modalities from Image Captions](https://mlsanity.com/paper/2204.00679/) (2022)
- [TSP: Temporally-Sensitive Pretraining of Video Encoders for Localization Tasks](https://mlsanity.com/paper/2011.11479/) (2020)

## Built on by (most cited first, 30 of 190)

- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling](https://mlsanity.com/paper/2501.00574/) (2024)
- [CogVLM2: Visual Language Models for Image and Video Understanding](https://mlsanity.com/paper/2408.16500/) (2024)
- [InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling](https://mlsanity.com/paper/2501.12386/) (2025)
- [Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding](https://mlsanity.com/paper/2503.13377/) (2025)
- [Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning](https://mlsanity.com/paper/2402.11435/) (2024)
- [Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding](https://mlsanity.com/paper/2403.09626/) (2024)
- [TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning](https://mlsanity.com/paper/2410.19702/) (2024)
- [TRACE: Temporal Grounding Video LLM via Causal Event Modeling](https://mlsanity.com/paper/2410.05643/) (2024)
- [E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding](https://mlsanity.com/paper/2409.18111/) (2024)
- [TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability](https://mlsanity.com/paper/2411.18211/) (2024)
- [Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization](https://mlsanity.com/paper/2601.12993/) (2026)
- [Number it: Temporal Grounding Videos like Flipping Manga](https://mlsanity.com/paper/2411.10332/) (2024)
- [VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception](https://mlsanity.com/paper/2509.21100/) (2025)
- [Grounded Question-Answering in Long Egocentric Videos](https://mlsanity.com/paper/2312.06505/) (2023)
- [VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation](https://mlsanity.com/paper/2408.16730/) (2024)
- [ChatVTG: Video Temporal Grounding via Chat with Video Dialogue Large Language Models](https://mlsanity.com/paper/2410.12813/) (2024)
- [Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model](https://mlsanity.com/paper/2312.12423/) (2023)
- [ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos](https://mlsanity.com/paper/2406.19392/) (2024)
- [LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos](https://mlsanity.com/paper/2411.19772/) (2024)
- [V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning](https://mlsanity.com/paper/2404.12353/) (2024)
- [Uncovering What, Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomaly](https://mlsanity.com/paper/2405.00181/) (2024)
- [Grounded Multi-Hop VideoQA in Long-Form Egocentric Videos](https://mlsanity.com/paper/2408.14469/) (2024)
- [MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos](https://mlsanity.com/paper/2406.08407/) (2024)
- [VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning](https://mlsanity.com/paper/2503.13444/) (2025)
- [Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillation](https://mlsanity.com/paper/2412.01694/) (2024)
- [SnAG: Scalable and Accurate Video Grounding](https://mlsanity.com/paper/2404.02257/) (2024)
- [R2-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding](https://mlsanity.com/paper/2404.00801/) (2024)
- [Correlation-Guided Query-Dependency Calibration for Video Temporal Grounding](https://mlsanity.com/paper/2311.08835/) (2023)

## Abstract (arXiv)

Video Temporal Grounding (VTG), which aims to ground target clips from videos (such as consecutive intervals or disjoint shots) according to custom language queries (e.g., sentences or words), is key for video browsing on social media. Most methods in this direction develop taskspecific models that are trained with type-specific labels, such as moment retrieval (time interval) and highlight detection (worthiness curve), which limits their abilities to generalize to various VTG tasks and labels. In this paper, we propose to Unify the diverse VTG labels and tasks, dubbed UniVTG, along three directions: Firstly, we revisit a wide range of VTG labels and tasks and define a unified formulation. Based on this, we develop data annotation schemes to create scalable pseudo supervision. Secondly, we develop an effective and flexible grounding model capable of addressing each task and making full use of each label. Lastly, thanks to the unified framework, we are able to unlock temporal grounding pretraining from large-scale diverse labels and develop stronger grounding abilities e.g., zero-shot grounding. Extensive experiments on three tasks (moment retrieval, highlight detection and video summarization) across seven datasets (QVHighlights, Charades-STA, TACoS, Ego4D, YouTube Highlights, TVSum, and QFVS) demonstrate the effectiveness and flexibility of our proposed framework. The codes are available at https://github.com/showlab/UniVTG.