# Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing

arXiv: https://arxiv.org/abs/2610.13823 · HTML: https://mlsanity.com/paper/2610.13823/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 57 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 24 of 24)

- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism](https://mlsanity.com/paper/2407.10457/) (2024)
- [Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes](https://mlsanity.com/paper/2304.09433/) (2023)
- [A Declarative System for Optimizing AI Workloads](https://mlsanity.com/paper/2405.14696/) (2024)
- [Optimizing LLM Queries in Relational Data Analytics Workloads](https://mlsanity.com/paper/2403.05821/) (2024)
- [UQE: A Query Engine for Unstructured Databases](https://mlsanity.com/paper/2407.09522/) (2024)
- [Approximate Selection with Guarantees using Proxies](https://mlsanity.com/paper/2004.00827/) (2020)
- [Beyond Quacking: Deep Integration of Language Models and RAG into DuckDB](https://mlsanity.com/paper/2504.01157/) (2025)
- [Abacus: A Cost-Based Optimizer for Semantic Operator Systems](https://mlsanity.com/paper/2505.14661/) (2025)
- [Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees](https://mlsanity.com/paper/2509.02896/) (2025)
- [Featurized-Decomposition Join: Low-Cost Semantic Joins with Guarantees](https://mlsanity.com/paper/2512.05399/) (2025)
- [Larch: Learned Query Optimization for Semantic Predicates](https://mlsanity.com/paper/2606.07923/) (2026)
- [Sema: A High-performance System for LLM-based Semantic Query Processing](https://mlsanity.com/paper/2603.11622/) (2026)
- [Unstructured Data Analysis using LLMs: A Comprehensive Benchmark](https://mlsanity.com/paper/2510.27119/) (2025)
- [Large Language Model-Enhanced Relational Operators: Taxonomy, Benchmark, and Analysis](https://mlsanity.com/paper/2603.02537/) (2026)
- [Accelerating Approximate Analytical Join Queries over Unstructured Data with Statistical Guarantees](https://mlsanity.com/paper/2603.16153/) (2026)
- [PLOP: Cost-Based Placement of Semantic Operators in Hybrid Query Plans](https://mlsanity.com/paper/2604.09944/) (2026)
- [Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method](https://mlsanity.com/paper/2606.08090/) (2026)
- [CADENZA: Compiling Natural-Language Intent into Task-Specific Operator DAGs for Semantic Query Processing](https://mlsanity.com/paper/2606.29151/) (2026)
- [Kalypso: Relational LLM Serving](https://mlsanity.com/paper/2607.23815/) (2026)
- [From Interpretation to Compilation: A Compilation-Based Execution Engine for Semantic Operator Systems](https://mlsanity.com/paper/2608.06677/) (2026)
- [ListK: Semantic ORDER BY and LIMIT K with Listwise Prompting](https://mlsanity.com/paper/2603.17223/) (2026)
- [The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI Operators](https://mlsanity.com/paper/2609.21133/) (2026)

## Abstract (arXiv)

Semantic query engines ask a large language model (LLM) to judge individual rows, and benchmarks compare these engines on the quality of a few executions at temperature 0. We measure how much another execution changes. We re-execute the text and tabular workloads of SemBench, LRO-Bench, and UDA-Bench through their own harnesses, log every one of 1.2 million model calls, and reanalyze SemBench's published executions. At temperature 0, between 0.07% and 10.2% of rows change their operator output over five executions, depending on the model, and up to 24.0% on one task. Plans amplify these changes exponentially in the number of rows an answer exposes, as a possible-worlds model predicts: on UDA-Bench, a few percent of extracted values change, but about two thirds of the answers do. In SemBench's published results, 21% of text results change quality between executions, including on engines whose plans are deterministic. Several per-query system comparisons, and 16 of 28 tests of LRO-Bench's implementation recommendations, are not settled by the executions the benchmarks run. Changing rows are rows on which models disagree with each other; identical requests within one execution disagree as often as across executions; and one execution's decision margin or reasoning length predicts which rows change. Voting reduces the variance but does not remove it, and a calibrated cascade keeps its guarantee while the rows it returns change. We derive reporting rules for benchmarks and design rules for engines.