# Abacus: A Cost-Based Optimizer for Semantic Operator Systems

arXiv: https://arxiv.org/abs/2505.14661 · HTML: https://mlsanity.com/paper/2505.14661/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 23 of 23)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [MultiModalQA: Complex Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2104.06039/) (2021)
- [Finite-Time Analysis of Kernelised Contextual Bandits](https://mlsanity.com/paper/1309.6869/) (2013)
- [MileBench: Benchmarking MLLMs in Long Context](https://mlsanity.com/paper/2404.18532/) (2024)
- [Archon: An Architecture Search Framework for Inference-Time Techniques](https://mlsanity.com/paper/2409.15254/) (2024)
- [DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing](https://mlsanity.com/paper/2410.12189/) (2024)
- [BlazeIt: Optimizing Declarative Aggregation and Limit Queries for Neural Network-Based Video Analytics](https://mlsanity.com/paper/1805.01046/) (2018)
- [Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together](https://mlsanity.com/paper/2407.10930/) (2024)
- [NoScope: Optimizing Neural Network Queries over Video at Scale](https://mlsanity.com/paper/1703.02529/) (2017)
- [Semantic Operators: A Declarative Model for Rich, AI-based Data Processing](https://mlsanity.com/paper/2407.11418/) (2024)
- [CAESURA: Language Models as Multi-Modal Query Planners](https://mlsanity.com/paper/2308.03424/) (2023)
- [Physical Representation-based Predicate Optimization for a Visual Analytics Database](https://mlsanity.com/paper/1806.04226/) (2018)
- [Accelerating Approximate Aggregation Queries with Expensive Predicates](https://mlsanity.com/paper/2108.06313/) (2021)
- [BioDEX: Large-Scale Biomedical Adverse Drug Event Extraction for Real-World Pharmacovigilance](https://mlsanity.com/paper/2305.13395/) (2023)
- [Extract-Transform-Load for Video Streams](https://mlsanity.com/paper/2310.04830/) (2023)
- [Accelerating Aggregation Queries on Unstructured Streams of Data](https://mlsanity.com/paper/2308.09157/) (2023)

## Built on by (most cited first, 13 of 13)

- [AgenticData: An Agentic Data Analytics System for Heterogeneous Data](https://mlsanity.com/paper/2508.05002/) (2025)
- [Beyond Relational: Semantic-Aware Multi-Modal Analytics with LLM-Native Query Optimization](https://mlsanity.com/paper/2511.19830/) (2025)
- [100x Cost & Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models](https://mlsanity.com/paper/2603.15970/) (2026)
- [CTIFoundry: An Agent-Native Corpus Scaffold for Cyber Threat Intelligence](https://mlsanity.com/paper/2608.18613/) (2026)
- [Both Ends Count! Just How Good are LLM Agents at "Text-to-Big SQL"?](https://mlsanity.com/paper/2602.21480/) (2026)
- [Evergreen: Efficient Claim Verification for Semantic Aggregates](https://mlsanity.com/paper/2604.26180/) (2026)
- [Opti-Q: A Constraint-Based Optimization Framework for Multi-LLM Question Planning](https://mlsanity.com/paper/2607.22621/) (2026)
- [Search over the Visual World: Persistent Visual Memory, Layered Indexes, and Source-Grounded Evidence](https://mlsanity.com/paper/2608.08075/) (2026)
- [Efficiently Linking Unstructured Data for Multi-step Reasoning](https://mlsanity.com/paper/2609.19491/) (2026)
- [KathDB-FAO: Synthesized Query Plans in a Multimodal DBMS](https://mlsanity.com/paper/2609.28761/) (2026)
- [Fail Loudly: An Auditable Runtime for Agentic Data Analysis](https://mlsanity.com/paper/2609.32528/) (2026)
- [Semantic Views: Certified Reuse of LLM Labels Across Semantic Queries](https://mlsanity.com/paper/2610.12768/) (2026)
- [Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing](https://mlsanity.com/paper/2610.13823/) (2026)

## Abstract (arXiv)

LLMs enable an exciting new class of data processing applications over large collections of unstructured documents. Several new programming frameworks have enabled developers to build these applications by composing them out of semantic operators: a declarative set of AI-powered data transformations with natural language specifications. These include LLM-powered maps, filters, joins, etc. used for document processing tasks such as information extraction, summarization, and more. While systems of semantic operators have achieved strong performance on benchmarks, they can be difficult to optimize. An optimizer for this setting must determine how to physically implement each semantic operator in a way that optimizes the system globally. Existing optimizers are limited in the number of optimizations they can apply, and most (if not all) cannot optimize system quality, cost, or latency subject to constraint(s) on the other dimensions. In this paper we present Abacus, an extensible, cost-based optimizer which searches for the best implementation of a semantic operator system given a (possibly constrained) optimization objective. Abacus estimates operator performance by leveraging a minimal set of validation examples, prior beliefs about operator performance, and/or an LLM judge. We evaluate Abacus on document processing workloads in the biomedical and legal domains (BioDEX; CUAD) and multi-modal question answering (MMQA). We demonstrate that, on-average, systems optimized by Abacus achieve 6.7%-39.4% better quality and are 10.8x cheaper and 3.4x faster than the next best system.