# TraVEL: Trajectory-Guided Video Embedding Learning for Driving-Video Retrieval

arXiv: https://arxiv.org/abs/2608.13495 · HTML: https://mlsanity.com/paper/2608.13495/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 13 Aug 2026.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking](https://mlsanity.com/paper/2601.04720/) (2026)
- [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://mlsanity.com/paper/2410.05160/) (2024)
- [VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents](https://mlsanity.com/paper/2507.04590/) (2025)
- [Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?](https://mlsanity.com/paper/2301.00184/) (2022)
- [VL-JEPA: Joint Embedding Predictive Architecture for Vision-language](https://mlsanity.com/paper/2512.10942/) (2025)
- [RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarios](https://mlsanity.com/paper/2312.13303/) (2023)
- [MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction](https://mlsanity.com/paper/2509.18095/) (2025)
- [AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2403.17373/) (2024)
- [DVGT: Driving Visual Geometry Transformer](https://mlsanity.com/paper/2512.16919/) (2025)
- [RefAV: Towards Planning-Centric Scenario Mining](https://mlsanity.com/paper/2505.20981/) (2025)
- [VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models](https://mlsanity.com/paper/2504.03970/) (2025)
- [Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions](https://mlsanity.com/paper/2503.05186/) (2025)
- [nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving](https://mlsanity.com/paper/2605.31572/) (2026)
- [VeRVE: Versatile Retrieval for Videos via Unified Embeddings](https://mlsanity.com/paper/2601.12193/) (2026)
- [The Case for Negative Data: From Crash Reports to Counterfactuals for Reasonable Driving](https://mlsanity.com/paper/2509.18626/) (2025)
- [VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving](https://mlsanity.com/paper/2606.12396/) (2026)
- [X-CoT: Explainable Text-to-Video Retrieval via LLM-based Chain-of-Thought Reasoning](https://mlsanity.com/paper/2509.21559/) (2025)
- [Rethinking Video-Text Understanding: Retrieval from Counterfactually Augmented Data](https://mlsanity.com/paper/2407.13094/) (2024)
- [Reconstruction Matters: Learning Geometry-Aligned BEV Representation through 3D Gaussian Splatting](https://mlsanity.com/paper/2603.19193/) (2026)
- [Driving Video Retrieval for Complex Queries with Structured Grounding](https://mlsanity.com/paper/2606.09109/) (2026)

## Abstract (arXiv)

Efficiently retrieving relevant clips from large-scale driving logs is essential for data curation, model development, and safety analysis. Structured and rule-based retrieval systems can explicitly target driving events, but typically require expert-defined rules, auxiliary data, and multi-stage perception pipelines. Multimodal embedding models offer a simpler and more efficient alternative by representing each video with a single searchable vector. However, general-purpose models often rely on shortcuts from static scene context and struggle to distinguish motion-centric events, such as turning left versus right or accelerating versus decelerating. In this work, we study how to adapt a general-purpose multimodal embedding model to driving-video retrieval. We first fine-tune Qwen3-VL-Embedding on paired clips and reasoning traces from nuReasoning using an InfoNCE objective. While this stage substantially improves overall retrieval, caption supervision alone remains insufficient for fine-grained motion understanding. We therefore introduce TraVEL (Trajectory-Guided Video Embedding Learning), a motion-aware fine-tuning framework that uses ego-trajectory similarity as a reward within Group Relative Policy Optimization. Trajectories serve only as privileged training supervision; retrieval still operates on single-vector video embeddings without ego poses, expert rules, or auxiliary perception outputs. We further construct a driving-video retrieval benchmark from nuReasoning. Experiments show that TraVEL improves motion-centric retrieval across model scales: relative to SFT, it raises longitudinal and lateral mAP by 9.8 and 4.7 points at 2B, with corresponding gains of 7.2 and 1.5 points at 8B. TraVEL thus combines physically grounded supervision with efficient embedding-based search.