# Unleash the Potential of CLIP for Video Highlight Detection

arXiv: https://arxiv.org/abs/2404.01745 · HTML: https://mlsanity.com/paper/2404.01745/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 20 references that are in the map.
- First posted on arXiv 2 Apr 2024.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [Zero-Shot Composed Image Retrieval with Textual Inversion](https://mlsanity.com/paper/2303.15247/) (2023)
- [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2303.13874/) (2023)
- [UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2203.12745/) (2022)
- [Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training](https://mlsanity.com/paper/2210.08773/) (2022)
- [DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training](https://mlsanity.com/paper/2303.03032/) (2023)
- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [To Click or Not To Click: Automatic Selection of Beautiful Thumbnails from Videos](https://mlsanity.com/paper/1609.01388/) (2016)

## Built on by (most cited first, 6 of 6)

- [TRACE: Temporal Grounding Video LLM via Causal Event Modeling](https://mlsanity.com/paper/2410.05643/) (2024)
- [TimeExpert: An Expert-Guided Video LLM for Video Temporal Grounding](https://mlsanity.com/paper/2508.01699/) (2025)
- [Moment of Untruth: Dealing with Negative Queries in Video Moment Retrieval](https://mlsanity.com/paper/2502.08544/) (2025)
- [Sparse-Dense Side-Tuner for efficient Video Temporal Grounding](https://mlsanity.com/paper/2507.07744/) (2025)
- [SGWIB:Sliced Gromov-Wasserstein Information Bottleneck for Video Highlight Detection](https://mlsanity.com/paper/2609.13966/) (2026)
- [Zero-shot video highlight detection based on text descriptions and synthetic images](https://mlsanity.com/paper/2609.14790/) (2026)

## Abstract (arXiv)

Multimodal and large language models (LLMs) have revolutionized the utilization of open-world knowledge, unlocking novel potentials across various tasks and applications. Among these domains, the video domain has notably benefited from their capabilities. In this paper, we present Highlight-CLIP (HL-CLIP), a method designed to excel in the video highlight detection task by leveraging the pre-trained knowledge embedded in multimodal models. By simply fine-tuning the multimodal encoder in combination with our innovative saliency pooling technique, we have achieved the state-of-the-art performance in the highlight detection task, the QVHighlight Benchmark, to the best of our knowledge.