# Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agents

arXiv: https://arxiv.org/abs/2609.22259 · HTML: https://mlsanity.com/paper/2609.22259/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 41 references that are in the map.
- First posted on arXiv 6 Sep 2026.

## Builds on (most cited first, 22 of 22)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [Defeating Prompt Injections by Design](https://mlsanity.com/paper/2503.18813/) (2025)
- [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://mlsanity.com/paper/2411.07763/) (2024)
- [With Little Power Comes Great Responsibility](https://mlsanity.com/paper/2010.06595/) (2020)
- [KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers](https://mlsanity.com/paper/2106.11455/) (2021)
- [Croissant: A Metadata Format for ML-Ready Datasets](https://mlsanity.com/paper/2403.19546/) (2024)
- [The Ladder: A Reliable Leaderboard for Machine Learning Competitions](https://mlsanity.com/paper/1502.04585/) (2015)
- [Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models](https://mlsanity.com/paper/2308.00675/) (2023)
- [The Leaderboard Illusion](https://mlsanity.com/paper/2504.20879/) (2025)
- [Can LLMs Follow Simple Rules?](https://mlsanity.com/paper/2311.04235/) (2023)
- [DABstep: Data Agent Benchmark for Multi-step Reasoning](https://mlsanity.com/paper/2506.23719/) (2025)
- [Text2SQL is Not Enough: Unifying AI and Databases with TAG](https://mlsanity.com/paper/2408.14717/) (2024)
- [A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases](https://mlsanity.com/paper/2311.07509/) (2023)
- [Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?](https://mlsanity.com/paper/2411.02093/) (2024)
- [Scaffold Effects on GAIA: A Controlled Comparison](https://mlsanity.com/paper/2606.08529/) (2026)
- [Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models](https://mlsanity.com/paper/2604.25149/) (2026)

## Abstract (arXiv)

Context layers, curated documentation that an analytics agent fetches at query time, produce large accuracy gains on text-to-SQL benchmarks. A with/without comparison cannot say which part of the layer does the work: the semantic content, the retrieval scaffolding that delivers it, or the pre-computed views that usually accompany it. We report a four-arm ablation on DABStep on four models that separates the three. The instrument is a data contract: a YAML artifact that carries a domain's semantics and the rules an agent's tools enforce. One arm empties every field of prose in the frozen contract while holding the tool surface, retrieval instruction, table allow-list and operation rules byte-for-byte fixed. Compiling the contract's own SQL expressions into views gives the ceiling a pre-computed layer would reach: gold on all 176 tasks it covers. Content dominates. It raises hard-task accuracy from 13.9% to 55.1%, 22.6% to 56.6%, 22.9% to 68.4% and 37.0% to 77.4%, beating the same knowledge pasted into the prompt on every model. Scaffolding without content is worth 0 to 5 points on two flash models and 14 to 15 on two frontier models. Against the compiled ceiling the contract arm's shortfall is a failure to derive, and it falls from 39 points to 5 with model capability. The contract beats the prompt because the rule it needs is one lookup away rather than buried in a long prompt: its SQL carries the fee semantics up to 98% of the time against the prompt arm's 4%, and at the lowest cost per correct answer on three of four models. For practitioners: semantics first, scaffolding second, pre-computed macros only where an agent demonstrably fails to derive. Ungoverned arms submitted 166 mutating statements; governed arms none. The gain is confined to the contract's domain. On one model the benchmark's own withheld golds grade the contract arm at 51.9% against 18.8% for the prompt baseline.