# A Declarative System for Optimizing AI Workloads

arXiv: https://arxiv.org/abs/2405.14696 · HTML: https://mlsanity.com/paper/2405.14696/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 61 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Accelerating LLM Inference with Staged Speculative Decoding](https://mlsanity.com/paper/2308.04623/) (2023)
- [Online Speculative Decoding](https://mlsanity.com/paper/2310.07177/) (2023)
- [Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes](https://mlsanity.com/paper/2304.09433/) (2023)
- [End-to-End Beam Retrieval for Multi-Hop Question Answering](https://mlsanity.com/paper/2308.08973/) (2023)
- [Validating Large Language Models with ReLM](https://mlsanity.com/paper/2211.15458/) (2022)
- [Optimizing LLM Queries in Relational Data Analytics Workloads](https://mlsanity.com/paper/2403.05821/) (2024)
- [CAESURA: Language Models as Multi-Modal Query Planners](https://mlsanity.com/paper/2308.03424/) (2023)
- [Physical Representation-based Predicate Optimization for a Visual Analytics Database](https://mlsanity.com/paper/1806.04226/) (2018)
- [Learning Complexity-Aware Cascades for Deep Pedestrian Detection](https://mlsanity.com/paper/1507.05348/) (2015)
- [Towards Accurate and Efficient Document Analytics with Large Language Models](https://mlsanity.com/paper/2405.04674/) (2024)
- [Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models](https://mlsanity.com/paper/2404.09529/) (2024)

## Built on by (most cited first, 30 of 37)

- [DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing](https://mlsanity.com/paper/2410.12189/) (2024)
- [In-depth Analysis of Graph-based RAG in a Unified Framework](https://mlsanity.com/paper/2503.04338/) (2025)
- [Semantic Operators: A Declarative Model for Rich, AI-based Data Processing](https://mlsanity.com/paper/2407.11418/) (2024)
- [Text2SQL is Not Enough: Unifying AI and Databases with TAG](https://mlsanity.com/paper/2408.14717/) (2024)
- [Atomix: Timely, Transactional Tool Use for Reliable Agentic Workflows](https://mlsanity.com/paper/2602.14849/) (2026)
- [The Design of an LLM-powered Unstructured Analytics System](https://mlsanity.com/paper/2409.00847/) (2024)
- [Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms](https://mlsanity.com/paper/2508.18298/) (2025)
- [MEBench: Benchmarking Large Language Models for Cross-Document Multi-Entity Question Answering](https://mlsanity.com/paper/2502.18993/) (2025)
- [SRAG: Structured Retrieval-Augmented Generation for Multi-Entity Question Answering over Wikipedia Graph](https://mlsanity.com/paper/2503.01346/) (2025)
- [Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective](https://mlsanity.com/paper/2603.16104/) (2026)
- [Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AI](https://mlsanity.com/paper/2504.08148/) (2025)
- [Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees](https://mlsanity.com/paper/2509.02896/) (2025)
- [Multi-Modal Data Exploration via Language Agents](https://mlsanity.com/paper/2412.18428/) (2024)
- [Visual Template Inference for Data Extraction from Documents](https://mlsanity.com/paper/2501.06659/) (2025)
- [Towards Resource-Efficient Compound AI Systems](https://mlsanity.com/paper/2501.16634/) (2025)
- [From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems](https://mlsanity.com/paper/2506.04565/) (2025)
- [Batch Query Processing and Optimization for Agentic Workflows](https://mlsanity.com/paper/2509.02121/) (2025)
- [AnyMatch -- Efficient Zero-Shot Entity Matching with a Small Language Model](https://mlsanity.com/paper/2409.04073/) (2024)
- [PalimpChat: Declarative and Interactive AI analytics](https://mlsanity.com/paper/2502.03368/) (2025)
- [LLM+KG@VLDB'24 Workshop Summary](https://mlsanity.com/paper/2410.01978/) (2024)
- [Variable Extraction for Model Recovery in Scientific Literature](https://mlsanity.com/paper/2411.14569/) (2024)
- [ScaleDoc: Scaling LLM-based Predicates over Large Document Collections](https://mlsanity.com/paper/2509.12610/) (2025)
- [Streaming Model Cascades for Semantic SQL](https://mlsanity.com/paper/2604.00660/) (2026)
- [Research Challenges in Relational Database Management Systems for LLM Queries](https://mlsanity.com/paper/2508.20912/) (2025)
- [LLM-Powered Proactive Data Systems](https://mlsanity.com/paper/2502.13016/) (2025)
- [TAIJI: MCP-based Multi-Modal Data Analytics on Data Lakes](https://mlsanity.com/paper/2505.11270/) (2025)
- [DRAMA: Unifying Data Retrieval and Analysis for Open-Domain Analytic Queries](https://mlsanity.com/paper/2510.27238/) (2025)
- [Blue Data Intelligence Layer: Streaming Data and Agents for Multi-source Multi-modal Data-Centric Applications](https://mlsanity.com/paper/2604.15233/) (2026)
- [Rank It, Then Ask It: Input Reranking for Maximizing the Performance of LLMs on Symmetric Tasks](https://mlsanity.com/paper/2412.00546/) (2024)
- [Effectiveness of Prompt Optimization in NL2SQL Systems](https://mlsanity.com/paper/2505.20591/) (2025)

## Abstract (arXiv)

A long-standing goal of data management systems has been to build systems which can compute quantitative insights over large corpora of unstructured data in a cost-effective manner. Until recently, it was difficult and expensive to extract facts from company documents, data from scientific papers, or metrics from image and video corpora. Today's models can accomplish these tasks with high accuracy. However, a programmer who wants to answer a substantive AI-powered query must orchestrate large numbers of models, prompts, and data operations. For even a single query, the programmer has to make a vast number of decisions such as the choice of model, the right inference method, the most cost-effective inference hardware, the ideal prompt design, and so on. The optimal set of decisions can change as the query changes and as the rapidly-evolving technical landscape shifts. In this paper we present Palimpzest, a system that enables anyone to process AI-powered analytical queries simply by defining them in a declarative language. The system uses its cost optimization framework to implement the query plan with the best trade-offs between runtime, financial cost, and output data quality. We describe the workload of AI-powered analytics tasks, the optimization methods that Palimpzest uses, and the prototype system itself. We evaluate Palimpzest on tasks in Legal Discovery, Real Estate Search, and Medical Schema Matching. We show that even our simple prototype offers a range of appealing plans, including one that is 3.3x faster and 2.9x cheaper than the baseline method, while also offering better data quality. With parallelism enabled, Palimpzest can produce plans with up to a 90.3x speedup at 9.1x lower cost relative to a single-threaded GPT-4 baseline, while obtaining an F1-score within 83.5% of the baseline. These require no additional work by the user.