# V-Agent: An Interactive Video Search System Using Vision-Language Models

arXiv: https://arxiv.org/abs/2512.16925 · HTML: https://mlsanity.com/paper/2512.16925/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 4 Nov 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://mlsanity.com/paper/2410.05160/) (2024)
- [CLIP2Video: Mastering Video-Text Retrieval via Image CLIP](https://mlsanity.com/paper/2106.11097/) (2021)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [GME: Improving Universal Multimodal Retrieval by Multimodal LLMs](https://mlsanity.com/paper/2412.16855/) (2024)
- [VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset](https://mlsanity.com/paper/2305.18500/) (2023)
- [MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs](https://mlsanity.com/paper/2411.02571/) (2024)
- [Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward](https://mlsanity.com/paper/2404.01258/) (2024)
- [VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset](https://mlsanity.com/paper/2304.08345/) (2023)
- [Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations](https://mlsanity.com/paper/2211.11427/) (2022)
- [Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages](https://mlsanity.com/paper/2310.04799/) (2023)
- [TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy](https://mlsanity.com/paper/2406.11678/) (2024)
- [Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models](https://mlsanity.com/paper/2201.08471/) (2022)
- [Neural Approaches to Multilingual Information Retrieval](https://mlsanity.com/paper/2209.01335/) (2022)
- [MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval](https://mlsanity.com/paper/2410.11619/) (2024)
- [Synergistic Interplay between Search and Large Language Models for Information Retrieval](https://mlsanity.com/paper/2305.07402/) (2023)
- [Engineering Conversational Search Systems: A Review of Applications, Architectures, and Functional Components](https://mlsanity.com/paper/2407.00997/) (2024)

## Built on by (most cited first, 3 of 3)

- [Beyond Closed-Pool Video Retrieval: A Benchmark and Agent Framework for Real-World Video Search and Moment Localization](https://mlsanity.com/paper/2602.10159/) (2026)
- [Less Detail, Better Answers: Degradation-Driven Prompting for VQA](https://mlsanity.com/paper/2604.04838/) (2026)
- [Agentic Video Understanding: A Survey](https://mlsanity.com/paper/2609.31713/) (2026)

## Abstract (arXiv)

We introduce V-Agent, a novel multi-agent platform designed for advanced video search and interactive user-system conversations. By fine-tuning a vision-language model (VLM) with a small video preference dataset and enhancing it with a retrieval vector from an image-text retrieval model, we overcome the limitations of traditional text-based retrieval systems in multimodal scenarios. The VLM-based retrieval model independently embeds video frames and audio transcriptions from an automatic speech recognition (ASR) module into a shared multimodal representation space, enabling V-Agent to interpret both visual and spoken content for context-aware video search. This system consists of three agents-a routing agent, a search agent, and a chat agent-that work collaboratively to address user intents by refining search outputs and communicating with users. The search agent utilizes the VLM-based retrieval model together with an additional re-ranking module to further enhance video retrieval quality. Our proposed framework demonstrates state-of-the-art zero-shot performance on the MultiVENT 2.0 benchmark, highlighting its potential for both academic research and real-world applications. The retrieval model and demo videos are available at https://huggingface.co/NCSOFT/multimodal-embedding.