# CAESURA: Language Models as Multi-Modal Query Planners

arXiv: https://arxiv.org/abs/2308.03424 · HTML: https://mlsanity.com/paper/2308.03424/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2023.

## Built on by (most cited first, 20 of 20)

- [A Declarative System for Optimizing AI Workloads](https://mlsanity.com/paper/2405.14696/) (2024)
- [AgenticData: An Agentic Data Analytics System for Heterogeneous Data](https://mlsanity.com/paper/2508.05002/) (2025)
- [The Design of an LLM-powered Unstructured Analytics System](https://mlsanity.com/paper/2409.00847/) (2024)
- [Abacus: A Cost-Based Optimizer for Semantic Operator Systems](https://mlsanity.com/paper/2505.14661/) (2025)
- [SemBench: A Benchmark for Semantic Query Processing Engines](https://mlsanity.com/paper/2511.01716/) (2025)
- [Multi-Modal Data Exploration via Language Agents](https://mlsanity.com/paper/2412.18428/) (2024)
- [Deep Research is the New Analytics System: Towards Building the Runtime for AI-Driven Analytics](https://mlsanity.com/paper/2509.02751/) (2025)
- [Beyond Relational: Semantic-Aware Multi-Modal Analytics with LLM-Native Query Optimization](https://mlsanity.com/paper/2511.19830/) (2025)
- [LaPuda: LLM-Enabled Policy-Based Query Optimizer for Multi-modal Data](https://mlsanity.com/paper/2403.13597/) (2024)
- [HCT-QA: A Benchmark for Question Answering on Human-Centric Tables](https://mlsanity.com/paper/2504.20047/) (2025)
- [TAIJI: MCP-based Multi-Modal Data Analytics on Data Lakes](https://mlsanity.com/paper/2505.11270/) (2025)
- [Implementing Semantic Join Operators Efficiently](https://mlsanity.com/paper/2510.08489/) (2025)
- [Memento: Towards Proactive Visualization of Everyday Memories with Personal Wearable AR Assistant](https://mlsanity.com/paper/2601.17622/) (2026)
- [Semantic Integrity Constraints: Declarative Guardrails for AI-Augmented Data Processing Systems](https://mlsanity.com/paper/2503.00600/) (2025)
- [Evergreen: Efficient Claim Verification for Semantic Aggregates](https://mlsanity.com/paper/2604.26180/) (2026)
- [Benchmarking Text-to-SQL under Role-Based Access Control](https://mlsanity.com/paper/2607.22115/) (2026)
- [Concord: A Video Relational Algebra for Cross-Modal Query Optimization](https://mlsanity.com/paper/2609.05756/) (2026)
- [KathDB-FAO: Synthesized Query Plans in a Multimodal DBMS](https://mlsanity.com/paper/2609.28761/) (2026)
- [WeaveData: A Multimodal Data Analysis System with Self-Critiquing and Self-Evolving LLM Plans](https://mlsanity.com/paper/2609.34764/) (2026)
- [When Plans Change Answers: Formalizing Cost-Accuracy Optimization for Semantic Queries](https://mlsanity.com/paper/2610.08089/) (2026)

## Abstract (arXiv)

Traditional query planners translate SQL queries into query plans to be executed over relational data. However, it is impossible to query other data modalities, such as images, text, or video stored in modern data systems such as data lakes using these query planners. In this paper, we propose Language-Model-Driven Query Planning, a new paradigm of query planning that uses Language Models to translate natural language queries into executable query plans. Different from relational query planners, the resulting query plans can contain complex operators that are able to process arbitrary modalities. As part of this paper, we present a first GPT-4 based prototype called CEASURA and show the general feasibility of this idea on two datasets. Finally, we discuss several ideas to improve the query planning capabilities of today's Language Models.