# Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2501.07972 · HTML: https://mlsanity.com/paper/2501.07972/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 24 of its 70 references that are in the map.
- First posted on arXiv 14 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 24 of 24)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [VTimeLLM: Empower LLM to Grasp Video Moments](https://mlsanity.com/paper/2311.18445/) (2023)
- [GroundingGPT:Language Enhanced Multi-modal Grounding Model](https://mlsanity.com/paper/2401.06071/) (2024)
- [Deconfounded Video Moment Retrieval with Causal Intervention](https://mlsanity.com/paper/2106.01534/) (2021)
- [TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones](https://mlsanity.com/paper/2312.16862/) (2023)
- [Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning](https://mlsanity.com/paper/2203.13049/) (2022)
- [A Decade's Battle on Dataset Bias: Are We There Yet?](https://mlsanity.com/paper/2403.08632/) (2024)
- [PRewrite: Prompt Rewriting with Reinforcement Learning](https://mlsanity.com/paper/2401.08189/) (2024)
- [Zero-Shot Video Moment Retrieval from Frozen Vision-Language Models](https://mlsanity.com/paper/2309.00661/) (2023)
- [EchoPrompt: Instructing the Model to Rephrase Queries for Improved In-context Learning](https://mlsanity.com/paper/2309.10687/) (2023)
- [SHE-Net: Syntax-Hierarchy-Enhanced Text-Video Retrieval](https://mlsanity.com/paper/2404.14066/) (2024)
- [MLP: Motion Label Prior for Temporal Sentence Localization in Untrimmed 3D Human Motions](https://mlsanity.com/paper/2404.13657/) (2024)

## Built on by (most cited first, 16 of 16)

- [Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook](https://mlsanity.com/paper/2503.18016/) (2025)
- [A Survey on Video Temporal Grounding with Multimodal Large Language Model](https://mlsanity.com/paper/2508.10922/) (2025)
- [HiFi-Portrait: Zero-shot Identity-preserved Portrait Generation with High-fidelity Multi-face Fusion](https://mlsanity.com/paper/2512.14542/) (2025)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [Point to Span: Zero-Shot Moment Retrieval for Navigating Unseen Hour-Long Videos](https://mlsanity.com/paper/2512.10363/) (2025)
- [MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning](https://mlsanity.com/paper/2508.21451/) (2025)
- [MLLMs Know When Before Speaking: Revealing and Recovering Temporal Grounding via Attention Cues](https://mlsanity.com/paper/2605.21954/) (2026)
- [DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding](https://mlsanity.com/paper/2607.00672/) (2026)
- [Temporal Grounding as a Learning Signal for Referring Video Object Segmentation](https://mlsanity.com/paper/2508.11955/) (2025)
- [Insight-A: Attribution-aware for Multimodal Misinformation Detection](https://mlsanity.com/paper/2511.21705/) (2025)
- [FaceSnap: Enhanced ID-fidelity Network for Tuning-free Portrait Customization](https://mlsanity.com/paper/2602.00627/) (2026)
- [Diff-PC: Identity-preserving and 3D-aware Controllable Diffusion for Zero-shot Portrait Customization](https://mlsanity.com/paper/2602.00639/) (2026)
- [Beyond Caption-Based Queries for Video Moment Retrieval](https://mlsanity.com/paper/2603.02363/) (2026)
- [Mitigating Modality and Language-Style Gaps for Zero-Shot Video Moment Retrieval](https://mlsanity.com/paper/2607.19027/) (2026)
- [Your VLM Already Knows When: Training-Free Temporal Grounding by Asking Yes or No](https://mlsanity.com/paper/2608.08315/) (2026)
- [Training-Free Temporal Abstraction for General Video Understanding](https://mlsanity.com/paper/2608.27929/) (2026)

## Abstract (arXiv)

The target of video moment retrieval (VMR) is predicting temporal spans within a video that semantically match a given linguistic query. Existing VMR methods based on multimodal large language models (MLLMs) overly rely on expensive high-quality datasets and time-consuming fine-tuning. Although some recent studies introduce a zero-shot setting to avoid fine-tuning, they overlook inherent language bias in the query, leading to erroneous localization. To tackle the aforementioned challenges, this paper proposes Moment-GPT, a tuning-free pipeline for zero-shot VMR utilizing frozen MLLMs. Specifically, we first employ LLaMA-3 to correct and rephrase the query to mitigate language bias. Subsequently, we design a span generator combined with MiniGPT-v2 to produce candidate spans adaptively. Finally, to leverage the video comprehension capabilities of MLLMs, we apply VideoChatGPT and span scorer to select the most appropriate spans. Our proposed method substantially outperforms the state-ofthe-art MLLM-based and zero-shot models on several public datasets, including QVHighlights, ActivityNet-Captions, and Charades-STA.