# LAVA: Language Driven Scalable and Versatile Traffic Video Analytics

arXiv: https://arxiv.org/abs/2507.19821 · HTML: https://mlsanity.com/paper/2507.19821/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 54 references that are in the map.
- First posted on arXiv 26 Jul 2025.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024)
- [BoT-SORT: Robust Associations Multi-Pedestrian Tracking](https://mlsanity.com/paper/2206.14651/) (2022)
- [Self-regulating Prompts: Foundational Model Adaptation without Forgetting](https://mlsanity.com/paper/2307.06948/) (2023)
- [FastReID: A Pytorch Toolbox for General Instance Re-identification](https://mlsanity.com/paper/2006.02631/) (2020)
- [Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension](https://mlsanity.com/paper/2411.13093/) (2024)
- [Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input](https://mlsanity.com/paper/2408.15542/) (2024)
- [VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning](https://mlsanity.com/paper/2505.12434/) (2025)
- [Focus: Querying Large Video Datasets with Low Latency and Low Cost](https://mlsanity.com/paper/1801.03493/) (2018)
- [BlazeIt: Optimizing Declarative Aggregation and Limit Queries for Neural Network-Based Video Analytics](https://mlsanity.com/paper/1805.01046/) (2018)
- [Approximate Thompson Sampling via Epistemic Neural Networks](https://mlsanity.com/paper/2302.09205/) (2023)
- [Scalable Neural Contextual Bandit for Recommender Systems](https://mlsanity.com/paper/2306.14834/) (2023)
- [Semantic Indexes for Machine Learning-based Queries over Unstructured Data](https://mlsanity.com/paper/2009.04540/) (2020)
- [Video Monitoring Queries](https://mlsanity.com/paper/2002.10537/) (2020)
- [Feature Re-Learning with Data Augmentation for Video Relevance Prediction](https://mlsanity.com/paper/2004.03815/) (2020)
- [SeeSaw: Interactive Ad-hoc Search Over Image Databases](https://mlsanity.com/paper/2208.06497/) (2022)
- [How Does Traffic Environment Quantitatively Affect the Autonomous Driving Prediction?](https://mlsanity.com/paper/2301.04414/) (2023)
- [Co-movement Pattern Mining from Videos](https://mlsanity.com/paper/2308.05370/) (2023)

## Abstract (arXiv)

In modern urban environments, camera networks generate massive amounts of operational footage -- reaching petabytes each day -- making scalable video analytics essential for efficient processing. Many existing approaches adopt an SQL-based paradigm for querying such large-scale video databases; however, this constrains queries to rigid patterns with predefined semantic categories, significantly limiting analytical flexibility. In this work, we explore a language-driven video analytics paradigm aimed at enabling flexible and efficient querying of high-volume video data driven by natural language. Particularly, we build textscLava, a system that accepts natural language queries and retrieves traffic targets across multiple levels of granularity and arbitrary categories. textscLava comprises three main components: 1) a multi-armed bandit-based efficient sampling method for video segment-level localization; 2) a video-specific open-world detection module for object-level retrieval; and 3) a long-term object trajectory extraction scheme for temporal object association, yielding complete trajectories for object-of-interests. To support comprehensive evaluation, we further develop a novel benchmark by providing diverse, semantically rich natural language predicates and fine-grained annotations for multiple videos. Experiments on this benchmark demonstrate that textscLava improves F1-scores for selection queries by 14%, reduces MPAE for aggregation queries by 0.39, and achieves top-k precision of 86%, while processing videos  9.6×  faster than the most accurate baseline. Our code and dataset are available at https://github.com/yuyanrui/LAVA.