# Streaming Model Cascades for Semantic SQL

arXiv: https://arxiv.org/abs/2604.00660 · HTML: https://mlsanity.com/paper/2604.00660/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 40 references that are in the map.
- First posted on arXiv 1 Apr 2026.

## Builds on (most cited first, 16 of 16)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [When Does Confidence-Based Cascade Deferral Suffice?](https://mlsanity.com/paper/2307.02764/) (2023)
- [A Declarative System for Optimizing AI Workloads](https://mlsanity.com/paper/2405.14696/) (2024)
- [UQE: A Query Engine for Unstructured Databases](https://mlsanity.com/paper/2407.09522/) (2024)
- [Online Cascade Learning for Efficient Inference over Streams](https://mlsanity.com/paper/2402.04513/) (2024)
- [Approximate Selection with Guarantees using Proxies](https://mlsanity.com/paper/2004.00827/) (2020)
- [Cascade-Aware Training of Language Models](https://mlsanity.com/paper/2406.00060/) (2024)
- [Cortex AISQL: A Production SQL Engine for Unstructured Data](https://mlsanity.com/paper/2511.07663/) (2025)
- [Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees](https://mlsanity.com/paper/2509.02896/) (2025)
- [Spline-Based Probability Calibration](https://mlsanity.com/paper/1809.07751/) (2018)
- [Efficiently Deploying LLMs with Controlled Risk](https://mlsanity.com/paper/2410.02173/) (2024)
- [100x Cost & Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models](https://mlsanity.com/paper/2603.15970/) (2026)

## Built on by (most cited first, 3 of 3)

- [Larch: Learned Query Optimization for Semantic Predicates](https://mlsanity.com/paper/2606.07923/) (2026)
- [Compositional Online Learning for Semantic Data Processing Systems](https://mlsanity.com/paper/2608.27244/) (2026)
- [Semantic Views: Certified Reuse of LLM Labels Across Semantic Queries](https://mlsanity.com/paper/2610.12768/) (2026)

## Abstract (arXiv)

Modern data warehouses extend SQL with semantic operators that invoke large language models on each qualifying row, making per-row inference orders of magnitude more expensive than traditional SQL. Model cascades reduce this cost by routing most rows through a fast proxy model and delegating uncertain cases to an expensive oracle. Prior SUPG-style cascades, however, require a global proxy-score pass that is itself an LLM-inference workload and blocks output in pipelined query engines. They also target either precision or recall and cannot serve workloads that need both. We formalize the cascade routing problem for streaming semantic SQL with independent parallel workers and present two complementary algorithms within this model. SUPG-IT extends SUPG from single-pass, single-metric estimation to streaming execution by iteratively refining two thresholds as oracle labels accumulate across batches, and is the first streaming cascade with joint probabilistic guarantees on user-specified precision and recall at a chosen failure probability δ. GAMCAL replaces user-specified targets with a single tradeoff parameter α between classification error and oracle cost, and learns a monotone Generalized Additive Model that calibrates proxy scores to true-positive probabilities and supplies pointwise uncertainty for stochastic routing. On six classification, filtering, and join benchmarks evaluated in a production semantic SQL engine, both algorithms reach F1 ≥ 0.95 at their best operating points. GAMCAL also leads all six datasets at a 20% delegation budget and reaches F1 ≥ 0.95 with up to 58% fewer oracle calls than LOTUS's SUPG cascade. SUPG-IT attains the highest best-case F1, with a mean of 0.989 across the six datasets.