# VQPy: An Object-Oriented Approach to Modern Video Analytics

arXiv: https://arxiv.org/abs/2311.01623 · HTML: https://mlsanity.com/paper/2311.01623/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 3 Nov 2023.

## Builds on (most cited first, 18 of 18)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer](https://mlsanity.com/paper/2112.01838/) (2021)
- [Focus: Querying Large Video Datasets with Low Latency and Low Cost](https://mlsanity.com/paper/1801.03493/) (2018)
- [BlazeIt: Optimizing Declarative Aggregation and Limit Queries for Neural Network-Based Video Analytics](https://mlsanity.com/paper/1805.01046/) (2018)
- [CityFlow-NL: Tracking and Retrieval of Vehicles at City Scale by Natural Language Descriptions](https://mlsanity.com/paper/2101.04741/) (2021)
- [The 7th AI City Challenge](https://mlsanity.com/paper/2304.07500/) (2023)
- [Physical Representation-based Predicate Optimization for a Visual Analytics Database](https://mlsanity.com/paper/1806.04226/) (2018)
- [Llama: A Heterogeneous & Serverless Framework for Auto-Tuning Video Analytics Pipelines](https://mlsanity.com/paper/2102.01887/) (2021)
- [Rekall: Specifying Video Events using Compositions of Spatiotemporal Labels](https://mlsanity.com/paper/1910.02993/) (2019)
- [Optimizing Machine Learning Inference Queries with Correlative Proxy Models](https://mlsanity.com/paper/2201.00309/) (2022)
- [Serving and Optimizing Machine Learning Workflows on Heterogeneous Infrastructures](https://mlsanity.com/paper/2205.04713/) (2022)
- [Accelerating Approximate Aggregation Queries with Expensive Predicates](https://mlsanity.com/paper/2108.06313/) (2021)
- [THIA: Accelerating Video Analytics using Early Inference and Fine-Grained Query Planning](https://mlsanity.com/paper/2102.08481/) (2021)

## Built on by (most cited first, 1 of 1)

- [Empower Vision Applications with LoRA LMM](https://mlsanity.com/paper/2411.00915/) (2024)

## Abstract (arXiv)

Video analytics is widely used in contemporary systems and services. At the forefront of video analytics are video queries that users develop to find objects of particular interest. Building upon the insight that video objects (e.g., human, animals, cars, etc.), the center of video analytics, are similar in spirit to objects modeled by traditional object-oriented languages, we propose to develop an object-oriented approach to video analytics. This approach, named VQPy, consists of a frontendunicodex2015a Python variant with constructs that make it easy for users to express video objects and their interactionsunicodex2015as well as an extensible backend that can automatically construct and optimize pipelines based on video objects. We have implemented and open-sourced VQPy, which has been productized in Cisco as part of its DeepVision framework.