# Towards a Flexible Scale-out Framework for Efficient Visual Data Query Processing

arXiv: https://arxiv.org/abs/2402.03283 · HTML: https://mlsanity.com/paper/2402.03283/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 33 references that are in the map.
- First posted on arXiv 5 Feb 2024.

## Builds on (most cited first, 3 of 3)

- [BlazeIt: Optimizing Declarative Aggregation and Limit Queries for Neural Network-Based Video Analytics](https://mlsanity.com/paper/1805.01046/) (2018)
- [NoScope: Optimizing Neural Network Queries over Video at Scale](https://mlsanity.com/paper/1703.02529/) (2017)
- [SOFA: An Extensible Logical Optimizer for UDF-heavy Dataflows](https://mlsanity.com/paper/1311.6335/) (2013)

## Abstract (arXiv)

There is growing interest in visual data management systems that support queries with specialized operations ranging from resizing an image to running complex machine learning models. With a plethora of such operations, the basic need to receive query responses in minimal time takes a hit, especially when the client desires to run multiple such operations in a single query. Existing systems provide an ad-hoc approach where different solutions are clubbed together to provide an end-to-end visual data management system. Unlike such solutions, the Visual Data Management System (VDMS) natively executes queries with multiple operations, thus providing an end-to-end solution. However, a fixed subset of native operations and a synchronous threading architecture limit its generality and scalability. In this paper, we develop VDMS-Async that adds the capability to run user-defined operations with VDMS and execute operations within a query on a remote server. VDMS-Async utilizes an event-driven architecture to create an efficient pipeline for executing operations within a query. Our experiments have shown that VDMS-Async reduces the query execution time by 2-3X compared to existing state-of-the-art systems. Further, remote operations coupled with an event-driven architecture enables VDMS-Async to scale query execution time linearly with the addition of every new remote server. We demonstrate a 64X reduction in query execution time when adding 64 remote servers.