# Beyond the Harness: End-to-End Optimization of Context Artifacts for Enterprise Text-to-SQL

arXiv: https://arxiv.org/abs/2608.22830 · HTML: https://mlsanity.com/paper/2608.22830/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 31 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Builds on (most cited first, 27 of 27)

- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://mlsanity.com/paper/2304.11015/) (2023)
- [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://mlsanity.com/paper/1911.04942/) (2019)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [Automated Design of Agentic Systems](https://mlsanity.com/paper/2408.08435/) (2024)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [Meta-Harness: End-to-End Optimization of Model Harnesses](https://mlsanity.com/paper/2603.28052/) (2026)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [TextGrad: Automatic "Differentiation" via Text](https://mlsanity.com/paper/2406.07496/) (2024)
- [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://mlsanity.com/paper/2411.07763/) (2024)
- [Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation](https://mlsanity.com/paper/1905.08205/) (2019)
- [RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL](https://mlsanity.com/paper/2302.05965/) (2023)
- [LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench](https://mlsanity.com/paper/2409.13373/) (2024)
- [Robust Text-to-SQL Generation with Execution-Guided Decoding](https://mlsanity.com/paper/1807.03100/) (2018)
- [BEAVER: An Enterprise Benchmark for Text-to-SQL](https://mlsanity.com/paper/2409.02038/) (2024)
- [ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Exploration](https://mlsanity.com/paper/2502.00675/) (2025)
- [Evaluating Cross-Domain Text-to-SQL Models and Benchmarks](https://mlsanity.com/paper/2310.18538/) (2023)
- [AgentSM: Semantic Memory for Agentic Text-to-SQL](https://mlsanity.com/paper/2601.15709/) (2026)
- [FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmark](https://mlsanity.com/paper/2409.19014/) (2024)
- [Knowledge Base Construction for Knowledge-Augmented Text-to-SQL](https://mlsanity.com/paper/2505.22096/) (2025)
- [ETM: Modern Insights into Perspective on Text-to-SQL Evaluation in the Age of Large Language Models](https://mlsanity.com/paper/2407.07313/) (2024)
- [TailorSQL: An NL2SQL System Tailored to Your Query Workload](https://mlsanity.com/paper/2505.23039/) (2025)

## Abstract (arXiv)

Deploying LLMs for enterprise Text-to-SQL is bottlenecked less by the model than by what context reaches it: business logic spans thousands of tables, and no model can ingest a full catalog at once. We argue that the most effective place to intervene is therefore the knowledge-base context the model consumes, and that this context should be constructed from historical usage rather than tuned for as a fixed input. Using a query-DAG decomposition--the same family of intermediates that enterprise benchmarks like BEAVER annotate, here recovered from production SQL--we compare the value of oracle query graphs versus retrieved knowledge-base context. In this ablation, retrieved knowledge-base context provides the largest marginal improvement when added to the full oracle graph. Building on this, we optimize a distillation procedure that turns historical query profiles into reusable SQL reference cards. On a benchmark of 5176 production queries from a major online retailer, optimizing these context artifacts yields larger gains (~12--25% AST similarity) than optimizing the retrieval harness (~3--12%). On the public BEAVER benchmark, which lacks the production-usage signals available in our internal setting, the picture is more mixed: table cards alone perform about the same as raw historical SQL. The best optimized variant retrieves both cards and raw SQL, scoring 9.00% versus 6.33% (p-value 0.12) for the comparable baseline on a held-out N=300 subset, using retrieved context and harness changes but no agentic loop.