mlsanityData to 14 Oct 2026

Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agents

arXiv 2609.22259 ↗First posted 6 Sep 2026cs.CL cs.AI cs.DBcited by 0

Not on the map. Why?

Abstract (arXiv)

Context layers, curated documentation that an analytics agent fetches at query time, produce large accuracy gains on text-to-SQL benchmarks. A with/without comparison cannot say which part of the layer does the work: the semantic content, the retrieval scaffolding that delivers it, or the pre-computed views that usually accompany it. We report a four-arm ablation on DABStep on four models that separates the three. The instrument is a data contract: a YAML artifact that carries a domain's semantics and the rules an agent's tools enforce. One arm empties every field of prose in the frozen contract while holding the tool surface, retrieval instruction, table allow-list and operation rules byte-for-byte fixed. Compiling the contract's own SQL expressions into views gives the ceiling a pre-computed layer would reach: gold on all 176 tasks it covers. Content dominates. It raises hard-task accuracy from 13.9% to 55.1%, 22.6% to 56.6%, 22.9% to 68.4% and 37.0% to 77.4%, beating the same knowledge pasted into the prompt on every model. Scaffolding without content is worth 0 to 5 points on two flash models and 14 to 15 on two frontier models. Against the compiled ceiling the contract arm's shortfall is a failure to derive, and it falls from 39 points to 5 with model capability. The contract beats the prompt because the rule it needs is one lookup away rather than buried in a long prompt: its SQL carries the fee semantics up to 98% of the time against the prompt arm's 4%, and at the lowest cost per correct answer on three of four models. For practitioners: semantics first, scaffolding second, pre-computed macros only where an agent demonstrably fails to derive. Ungoverned arms submitted 166 mutating statements; governed arms none. The gain is confined to the contract's domain. On one model the benchmark's own withheld golds grade the contract arm at 51.9% against 18.8% for the prompt baseline.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 41 references that are in the map. First posted on arXiv 6 Sep 2026.

Builds on

22 of 41 references.

References · 22

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  4. 2018Datasheets for Datasetscited by 1,386
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  6. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  7. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  8. 2025Defeating Prompt Injections by Designcited by 297
  9. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
  10. 2020With Little Power Comes Great Responsibilitycited by 142
  11. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
  12. 2024Croissant: A Metadata Format for ML-Ready Datasetscited by 81
10 more
  1. 2015The Ladder: A Reliable Leaderboard for Machine Learning Competitionscited by 69
  2. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 68
  3. 2025The Leaderboard Illusioncited by 66
  4. 2023Can LLMs Follow Simple Rules?cited by 46
  5. 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 46
  6. 2024Text2SQL is Not Enough: Unifying AI and Databases with TAGcited by 34
  7. 2023A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databasescited by 25
  8. 2024Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?cited by 19
  9. 2026Scaffold Effects on GAIA: A Controlled Comparisoncited by 5
  10. 2026Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Modelscited by 3

Built on by

Not cited yet.