mlsanityData to 14 Oct 2026

Memory Architectures for Multi-Turn Text-to-SQL: A Benchmark and Empirical Study

arXiv 2605.26394 ↗First posted 25 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Multi-turn Text-to-SQL is central to enterprise analytics yet remains predominantly evaluated in single-turn settings. We introduce EnterpriseMem-Bench, a multi-turn Text-to-SQL benchmark of 300 sessions and 1,400 turns built programmatically from three enterprise domains (BIRD financial, SEC EDGAR, Northwind), with deterministic ground truth and per-turn memory-critical annotation. We evaluate five frontier models -- GPT-5 mini, GPT-5.2, Claude Sonnet 4.5, Sonnet 4.6, and Opus 4.6 -- across five memory conditions enabling a three-way ablation isolating working-memory window size, episodic retrieval, and semantic augmentation as independent effects. All Claude models are evaluated with extended thinking enabled to maintain parity with GPT reasoning models. We introduce the Memory Benefit Score (MBS) as a per-turn diagnostic metric. Four findings emerge: (1) stateless multi-turn Text-to-SQL collapses to zero execution accuracy by Turn 3 across all five models, even under reasoning; (2) memory-architecture complexity does not monotonically improve accuracy -- working memory dominates, and additional components produce model- and dataset-dependent effects from +14 to -16 percentage points; (3) Claude Sonnet 4.6 underperforms Sonnet 4.5 by 17-33pp on SEC EDGAR across conditions, a generational regression persisting under reasoning; (4) under reasoning, Claude error distributions become mono-modal -- every non-correct turn is a wrong-result error. We release the benchmark, agent, and evaluation code.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 13 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

11 of 13 references.

References · 11

  1. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  2. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  3. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  4. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  5. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  6. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  7. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 185
  8. 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 144
  9. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 120
  10. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 19
  11. 2024Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Typescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond the Harness: End-to-End Optimization of Context Artifacts for Enterprise Text-to-SQL
  2. 2025Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration
  3. 2025Text2Mem: A Unified Memory Operation Language for Memory Operating System
  4. 2026AgentSM: Semantic Memory for Agentic Text-to-SQL
  5. 2024BEAVER: An Enterprise Benchmark for Text-to-SQL
  6. 2025Text-to-SQL as Dual-State Reasoning: Integrating Adaptive Context and Progressive Generation
  7. 2024Multi-Turn Interactions for Text-to-SQL with Large Language Models
  8. 2025BAPPA: Benchmarking Agents, Plans, and Pipelines for Automated Text-to-SQL Generation