# GPTSee: Enhancing Moment Retrieval and Highlight Detection via Description-Based Similarity Features

arXiv: https://arxiv.org/abs/2403.01437 · HTML: https://mlsanity.com/paper/2403.01437/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 3 Mar 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [MultiModal-GPT: A Vision and Language Model for Dialogue with Humans](https://mlsanity.com/paper/2305.04790/) (2023)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021)
- [UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2203.12745/) (2022)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [ExCL: Extractive Clip Localization Using Natural Language Descriptions](https://mlsanity.com/paper/1904.02755/) (2019)
- [MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformer](https://mlsanity.com/paper/2305.00355/) (2023)
- [Cross-category Video Highlight Detection via Set-based Learning](https://mlsanity.com/paper/2108.11770/) (2021)
- [Temporal Sentence Grounding in Videos: A Survey and Future Directions](https://mlsanity.com/paper/2201.08071/) (2022)
- [ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval](https://mlsanity.com/paper/2203.16778/) (2022)

## Built on by (most cited first, 6 of 6)

- [A Survey on Video Temporal Grounding with Multimodal Large Language Model](https://mlsanity.com/paper/2508.10922/) (2025)
- [VTG-GPT: Tuning-Free Zero-Shot Video Temporal Grounding with GPT](https://mlsanity.com/paper/2403.02076/) (2024)
- [Multi-modal Fusion and Query Refinement Network for Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2501.10692/) (2025)
- [FaceSnap: Enhanced ID-fidelity Network for Tuning-free Portrait Customization](https://mlsanity.com/paper/2602.00627/) (2026)
- [Diff-PC: Identity-preserving and 3D-aware Controllable Diffusion for Zero-shot Portrait Customization](https://mlsanity.com/paper/2602.00639/) (2026)
- [Mitigating Modality and Language-Style Gaps for Zero-Shot Video Moment Retrieval](https://mlsanity.com/paper/2607.19027/) (2026)

## Abstract (arXiv)

Moment retrieval (MR) and highlight detection (HD) aim to identify relevant moments and highlights in video from corresponding natural language query. Large language models (LLMs) have demonstrated proficiency in various computer vision tasks. However, existing methods for MR&HD have not yet been integrated with LLMs. In this letter, we propose a novel two-stage model that takes the output of LLMs as the input to the second-stage transformer encoder-decoder. First, MiniGPT-4 is employed to generate the detailed description of the video frame and rewrite the query statement, fed into the encoder as new features. Then, semantic similarity is computed between the generated description and the rewritten queries. Finally, continuous high-similarity video frames are converted into span anchors, serving as prior position information for the decoder. Experiments demonstrate that our approach achieves a state-of-the-art result, and by using only span anchors and similarity scores as outputs, positioning accuracy outperforms traditional methods, like Moment-DETR.