# UQE: A Query Engine for Unstructured Databases

arXiv: https://arxiv.org/abs/2407.09522 · HTML: https://mlsanity.com/paper/2407.09522/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 23 Jun 2024.

## Builds on (most cited first, 28 of 28)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://mlsanity.com/paper/2304.11015/) (2023)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [Binding Language Models in Symbolic Languages](https://mlsanity.com/paper/2210.02875/) (2022)
- [Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation](https://mlsanity.com/paper/1905.08205/) (2019)
- [Large Language Models are few(1)-shot Table Reasoners](https://mlsanity.com/paper/2210.06710/) (2022)
- [Prompting Is Programming: A Query Language for Large Language Models](https://mlsanity.com/paper/2212.06094/) (2022)
- [Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.00783/) (2021)
- [SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Task](https://mlsanity.com/paper/1810.05237/) (2018)
- [AudioMNIST: Exploring Explainable Artificial Intelligence for Audio Analysis on a Simple Benchmark](https://mlsanity.com/paper/1807.03418/) (2018)
- [Table-GPT: Table-tuned GPT for Diverse Table Tasks](https://mlsanity.com/paper/2310.09263/) (2023)
- [Neural Programmer: Inducing Latent Programs with Gradient Descent](https://mlsanity.com/paper/1511.04834/) (2015)
- [Optimizing LLM Queries in Relational Data Analytics Workloads](https://mlsanity.com/paper/2403.05821/) (2024)
- [Querying Large Language Models with SQL](https://mlsanity.com/paper/2304.00472/) (2023)
- [Neural Databases](https://mlsanity.com/paper/2010.06973/) (2020)
- [DocumentNet: Bridging the Data Gap in Document Pre-Training](https://mlsanity.com/paper/2306.08937/) (2023)
- [Neural Enquirer: Learning to Query Tables with Natural Language](https://mlsanity.com/paper/1512.00965/) (2015)

## Built on by (most cited first, 21 of 21)

- [Semantic Operators: A Declarative Model for Rich, AI-based Data Processing](https://mlsanity.com/paper/2407.11418/) (2024)
- [AgenticData: An Agentic Data Analytics System for Heterogeneous Data](https://mlsanity.com/paper/2508.05002/) (2025)
- [Cortex AISQL: A Production SQL Engine for Unstructured Data](https://mlsanity.com/paper/2511.07663/) (2025)
- [A Survey of Large Language Model-Based Generative AI for Text-to-SQL: Benchmarks, Applications, Use Cases, and Challenges](https://mlsanity.com/paper/2412.05208/) (2024)
- [Weaver: Interweaving SQL and LLM for Table Reasoning](https://mlsanity.com/paper/2505.18961/) (2025)
- [GateLens: A Reasoning-Enhanced LLM Agent for Automotive Software Release Analytics](https://mlsanity.com/paper/2503.21735/) (2025)
- [100x Cost & Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models](https://mlsanity.com/paper/2603.15970/) (2026)
- [Streaming Model Cascades for Semantic SQL](https://mlsanity.com/paper/2604.00660/) (2026)
- [Research Challenges in Relational Database Management Systems for LLM Queries](https://mlsanity.com/paper/2508.20912/) (2025)
- [Beyond Linear LLM Invocation: An Efficient and Effective Semantic Filter Paradigm](https://mlsanity.com/paper/2603.04799/) (2026)
- [Semantic Integrity Constraints: Declarative Guardrails for AI-Augmented Data Processing Systems](https://mlsanity.com/paper/2503.00600/) (2025)
- [SABER: A SQL-Compatible Semantic Document Processing System Based on Extended Relational Algebra](https://mlsanity.com/paper/2509.00277/) (2025)
- [Beyond Single-Modal Analytics: A Framework for Integrating Heterogeneous LLM-Based Query Systems for Multi-Modal Data](https://mlsanity.com/paper/2602.01701/) (2026)
- [Evergreen: Efficient Claim Verification for Semantic Aggregates](https://mlsanity.com/paper/2604.26180/) (2026)
- [A Query Engine for the Agents](https://mlsanity.com/paper/2605.27785/) (2026)
- [SAGE: A Unified Algebra and Self-Adaptive Execution for AI Functions in SQL](https://mlsanity.com/paper/2608.20630/) (2026)
- [Compositional Online Learning for Semantic Data Processing Systems](https://mlsanity.com/paper/2608.27244/) (2026)
- [Towards Reliable AI Data Scientists: Data Agents with Workflow Harnesses](https://mlsanity.com/paper/2609.35255/) (2026)
- [The Hidden States Cookbook: A Large-Scale Ablation Study for Noise-Robust Conversational Intent Classification in Industry](https://mlsanity.com/paper/2610.05394/) (2026)
- [Semantic Views: Certified Reuse of LLM Labels Across Semantic Queries](https://mlsanity.com/paper/2610.12768/) (2026)
- [Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing](https://mlsanity.com/paper/2610.13823/) (2026)

## Abstract (arXiv)

Analytics on structured data is a mature field with many successful methods. However, most real world data exists in unstructured form, such as images and conversations. We investigate the potential of Large Language Models (LLMs) to enable unstructured data analytics. In particular, we propose a new Universal Query Engine (UQE) that directly interrogates and draws insights from unstructured data collections. This engine accepts queries in a Universal Query Language (UQL), a dialect of SQL that provides full natural language flexibility in specifying conditions and operators. The new engine leverages the ability of LLMs to conduct analysis of unstructured data, while also allowing us to exploit advances in sampling and optimization techniques to achieve efficient and accurate query execution. In addition, we borrow techniques from classical compiler theory to better orchestrate the workflow between sampling methods and foundation model calls. We demonstrate the efficiency of UQE on data analytics across different modalities, including images, dialogs and reviews, across a range of useful query types, including conditional aggregation, semantic retrieval and abstraction aggregation.