# Adaptive Multi-Agent Reasoning for Text-to-Video Retrieval

arXiv: https://arxiv.org/abs/2602.19040 · HTML: https://mlsanity.com/paper/2602.19040/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 25 of its 45 references that are in the map.
- First posted on arXiv 2 Dec 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [Seed1.5-VL Technical Report](https://mlsanity.com/paper/2505.07062/) (2025)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.04840/) (2024)
- [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://mlsanity.com/paper/2003.00392/) (2020)
- [Dual Encoding for Zero-Example Video Retrieval](https://mlsanity.com/paper/1809.06181/) (2018)
- [Vision-Language Models Do Not Understand Negation](https://mlsanity.com/paper/2501.09425/) (2025)
- [Predicting Visual Features from Text for Image and Video Caption Retrieval](https://mlsanity.com/paper/1709.01362/) (2017)
- [Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval](https://mlsanity.com/paper/2201.09168/) (2022)
- [SEA: Sentence Encoder Assembly for Video Retrieval by Textual Queries](https://mlsanity.com/paper/2011.12091/) (2020)
- [Lightweight Attentional Feature Fusion: A New Baseline for Text-to-Video Retrieval](https://mlsanity.com/paper/2112.01832/) (2021)
- [TRECVID 2019: An Evaluation Campaign to Benchmark Video Activity Detection, Video Captioning and Matching, and Video Search & Retrieval](https://mlsanity.com/paper/2009.09984/) (2020)
- [Learn to Understand Negation in Video Retrieval](https://mlsanity.com/paper/2205.00132/) (2022)
- [Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions](https://mlsanity.com/paper/2503.05186/) (2025)
- [Learning Audio-guided Video Representation with Gated Attention for Video-Text Retrieval](https://mlsanity.com/paper/2504.02397/) (2025)
- [An overview on the evaluated video retrieval tasks at TRECVID 2022](https://mlsanity.com/paper/2306.13118/) (2023)
- [Interpretable Embedding for Ad-hoc Video Search](https://mlsanity.com/paper/2402.11812/) (2024)
- [(Un)likelihood Training for Interpretable Embedding](https://mlsanity.com/paper/2207.00282/) (2022)
- [Improving Interpretable Embeddings for Ad-hoc Video Search with Generative Captions and Multi-word Concept Bank](https://mlsanity.com/paper/2404.06173/) (2024)
- [Benchmarking for Domain-Specific LLMs: A Case Study on Academia and Beyond](https://mlsanity.com/paper/2508.07353/) (2025)
- [Multimodal LLM-based Query Paraphrasing for Video Search](https://mlsanity.com/paper/2407.12341/) (2024)

## Abstract (arXiv)

The rise of short-form video platforms and the emergence of multimodal large language models (MLLMs) have amplified the need for scalable, effective, zero-shot text-to-video retrieval systems. While recent advances in large-scale pretraining have improved zero-shot cross-modal alignment, existing methods still struggle with query-dependent temporal reasoning, limiting their effectiveness on complex queries involving temporal, logical, or causal relationships. To address these limitations, we propose an adaptive multi-agent retrieval framework that dynamically orchestrates specialized agents over multiple reasoning iterations based on the demands of each query. The framework includes: (1) a retrieval agent for scalable retrieval over large video corpora, (2) a reasoning agent for zero-shot contextual temporal reasoning, and (3) a query reformulation agent for refining ambiguous queries and recovering performance for those that degrade over iterations. These agents are dynamically coordinated by an orchestration agent, which leverages intermediate feedback and reasoning outcomes to guide execution. We also introduce a novel communication mechanism that incorporates retrieval-performance memory and historical reasoning traces to improve coordination and decision-making. Experiments on three TRECVid benchmarks spanning eight years show that our framework achieves a twofold improvement over CLIP4Clip and significantly outperforms state-of-the-art methods by a large margin.