# Reasoning Text-to-Video Retrieval via Digital Twin Video Representations and Large Language Models

arXiv: https://arxiv.org/abs/2511.12371 · HTML: https://mlsanity.com/paper/2511.12371/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 15 Nov 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 18 of 18)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding](https://mlsanity.com/paper/2410.17434/) (2024)
- [Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension](https://mlsanity.com/paper/2411.13093/) (2024)
- [VISA: Reasoning Video Object Segmentation via Large Language Models](https://mlsanity.com/paper/2407.11325/) (2024)
- [VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos](https://mlsanity.com/paper/2502.01549/) (2025)
- [VSE++: Improving Visual-Semantic Embeddings with Hard Negatives](https://mlsanity.com/paper/1707.05612/) (2017)
- [Reasoning Segmentation for Images and Videos: A Survey](https://mlsanity.com/paper/2505.18816/) (2025)
- [Online Reasoning Video Segmentation with Just-in-Time Digital Twins](https://mlsanity.com/paper/2503.21056/) (2025)
- [Position: Foundation Models Need Digital Twin Representations](https://mlsanity.com/paper/2505.03798/) (2025)
- [Long Video Understanding with Learnable Retrieval in Video-Language Models](https://mlsanity.com/paper/2312.04931/) (2023)
- [Operating Room Workflow Analysis via Reasoning Segmentation over Digital Twins](https://mlsanity.com/paper/2503.21054/) (2025)
- [RVTBench: A Benchmark for Visual Reasoning Tasks](https://mlsanity.com/paper/2505.11838/) (2025)
- [RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graph](https://mlsanity.com/paper/2505.03173/) (2025)
- [Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarks](https://mlsanity.com/paper/2210.05038/) (2022)

## Built on by (most cited first, 3 of 3)

- [Driving Video Retrieval for Complex Queries with Structured Grounding](https://mlsanity.com/paper/2606.09109/) (2026)
- [Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA](https://mlsanity.com/paper/2606.17279/) (2026)
- [Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins](https://mlsanity.com/paper/2606.17298/) (2026)

## Abstract (arXiv)

The goal of text-to-video retrieval is to search large databases for relevant videos based on text queries. Existing methods have progressed to handling explicit queries where the visual content of interest is described explicitly; however, they fail with implicit queries where identifying videos relevant to the query requires reasoning. We introduce reasoning text-to-video retrieval, a paradigm that extends traditional retrieval to process implicit queries through reasoning while providing object-level grounding masks that identify which entities satisfy the query conditions. Instead of relying on vision-language models directly, we propose representing video content as digital twins, i.e., structured scene representations that decompose salient objects through specialist vision models. This approach is beneficial because it enables large language models to reason directly over long-horizon video content without visual token compression. Specifically, our two-stage framework first performs compositional alignment between decomposed sub-queries and digital twin representations for candidate identification, then applies large language model-based reasoning with just-in-time refinement that invokes additional specialist models to address information gaps. We construct a benchmark of 447 manually created implicit queries with 135 videos (ReasonT2VBench-135) and another more challenging version of 1000 videos (ReasonT2VBench-1000). Our method achieves 81.2% R@1 on ReasonT2VBench-135, outperforming the strongest baseline by greater than 50 percentage points, and maintains 81.7% R@1 on the extended configuration while establishing state-of-the-art results in three conventional benchmarks (MSR-VTT, MSVD, and VATEX).