# Compositional Online Learning for Semantic Data Processing Systems

arXiv: https://arxiv.org/abs/2608.27244 · HTML: https://mlsanity.com/paper/2608.27244/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Builds on (most cited first, 14 of 14)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [ServerlessLLM: Low-Latency Serverless Inference for Large Language Models](https://mlsanity.com/paper/2401.14351/) (2024)
- [DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing](https://mlsanity.com/paper/2410.12189/) (2024)
- [UQE: A Query Engine for Unstructured Databases](https://mlsanity.com/paper/2407.09522/) (2024)
- [Approximate Selection with Guarantees using Proxies](https://mlsanity.com/paper/2004.00827/) (2020)
- [Cortex AISQL: A Production SQL Engine for Unstructured Data](https://mlsanity.com/paper/2511.07663/) (2025)
- [QUEST: Query Optimization in Unstructured Document Analysis](https://mlsanity.com/paper/2507.06515/) (2025)
- [Larch: Learned Query Optimization for Semantic Predicates](https://mlsanity.com/paper/2606.07923/) (2026)
- [Hybrid Querying Over Relational Databases and Large Language Models](https://mlsanity.com/paper/2408.00884/) (2024)
- [100x Cost & Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models](https://mlsanity.com/paper/2603.15970/) (2026)
- [Streaming Model Cascades for Semantic SQL](https://mlsanity.com/paper/2604.00660/) (2026)
- [SEMA-SQL: Beyond Traditional Relational Querying with Large Language Models](https://mlsanity.com/paper/2604.23477/) (2026)
- [Access Paths for Efficient Ordering with Large Language Models](https://mlsanity.com/paper/2509.00303/) (2025)

## Abstract (arXiv)

An LLM call in a semantic data processing system is expensive enough to dominate query cost, yet slow enough to hide a CPU-side learner's update behind its round-trip. In production, LLM compute accounts for 80-90% of query cost, and each call costs 105-107× a relational predicate. The latency window inverts a design constraint of classical adaptive query processing, where online learners had to stay lightweight to avoid dominating the predicates they optimize. At LLM latency, per-call gradient steps and per-batch threshold solves fit inside the round-trip. We develop compositional online learning at the LLM call boundary: a framework for combining online-learning components in semantic data processing systems. Each component makes execution-time decisions and refines its learned artifacts online. The design space spans two axes, decision granularity and learner update cadence, and the components share a single learning pattern that hides each trainer step inside the next LLM round-trip. A production case study in Cortex AISQL composes three components: a memoization layer, an online per-call filter-ordering learner, and an online per-batch cascade-routing learner. A conditional cost decomposition assigns each learning component to a distinct factor of per-row LLM cost. Under independence, the two learning components compose multiplicatively to an 11.4× upper bound on a representative conjunction-filter workload. Self-selection at the cascade boundary, sample-budget shrinkage, and selectivity-estimation drift reduce it to a realistic figure near 8×.