mlsanityData to 14 Oct 2026

DSABench: Benchmarking Source Discovery, Coordination, and Execution for Multi-Database Agents

arXiv 2610.14700 ↗First posted 10 Oct 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Most text-to-SQL benchmarks provide the target database. In a large data catalog, however, choosing the sources is part of the problem: an agent must find every database needed for a request, determine how their results fit together, and execute the resulting workflow. We present DSABench to evaluate these capabilities jointly. Its 815 human-reviewed tasks draw on Spider, BIRD, and LiveSQLBench; 618 require several databases to contribute, and 105 allow more than one interpretation of the required sources. Executable contracts define source participation, cross-source dependencies, and source-local execution, while RouteEnv lets agents inspect public catalog evidence without revealing the gold annotations. In our routing-only study across seven model backends, the highest RA-Set score is 60.55%, compared with a highest RA-Graph score of 34.66% when coordination requirements are included. The gap becomes more pronounced when execution is required. Across four backends, median non-privileged Route+Exec-Graph success ranges from 6.69% to 7.98% without clarification and from 11.44% to 13.25% with one clarification; the best observed clarified configuration reaches 16.99%. On ambiguous tasks, clarification improves exact branch-source recovery by 39.8-43.2 percentage points. These findings illustrate why retrieving plausible databases or producing correct local SQL is not enough: an agent must carry the right information through the entire multi-database workflow.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

29 of 37 references.

References · 29

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  3. 2018Datasheets for Datasetscited by 1,386
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  7. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  8. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  9. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  10. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 424
  11. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 411
  12. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
17 more
  1. 2022Binding Language Models in Symbolic Languagescited by 228
  2. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 225
  3. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 195
  4. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 185
  5. 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 169
  6. 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 144
  7. 2024CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQLcited by 138
  8. 2018Improving Text-to-SQL Evaluation Methodologycited by 135
  9. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 126
  10. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 125
  11. 2023Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustnesscited by 53
  12. 2025LinkAlign: Scalable Schema Linking for Real-World Large-Scale Multi-Database Text-to-SQLcited by 29
  13. 2018Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sourcescited by 25
  14. 2024PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queriescited by 19
  15. 2023DBCopilot: Natural Language Querying over Massive Databases via Schema Routingcited by 10
  16. 2025DBRouting: Routing End User Queries to Databases for Answerabilitycited by 2
  17. 2026MDB-Link: Hierarchical Schema Linking for Multi-Database Text-to-SQLcited by 1

Built on by

Not cited yet.