ModularSQL: A Runtime Guardrail for the Multiplicity Blind Spot in Text-to-SQL
On the map: LLM safety and security → Safety alignment, refusal and guardrails
Abstract (arXiv)
Text-to-SQL systems are increasingly deployed on production databases, where queries that pass benchmark evaluation can still produce results that distort downstream workflows. Standard set-based execution accuracy (Set-EX) collapses duplicate rows and can therefore miss multiplicity errors, including missing DISTINCT, inflated aggregates, and Cartesian-style join explosions. We call this the Multiplicity Blind Spot (MBS) and introduce Multiset-EX, a multiplicity-preserving evaluation criterion that exposes such failures. Across released DeepEye-SQL artifacts from three backbones (Qwen2.5-Coder-32B, Qwen3-Coder-30B-A3B, and Gemma-3-27B) on executable BIRD-Dev N=1532, we find a consistent 5.81--6.79 pp gap between Set-EX and Multiset-EX. The gap is not specific to DeepEye-SQL: it persists on released DAIL-SQL+GPT-4 (5.22 pp) and BIRD GPT-3.5-turbo (3.39 pp) predictions. We further introduce ModularSQL, a lightweight post-selection runtime guardrail that probes executed results for multiplicity anomalies and applies deterministic patches or low-cost LLM rescue only to flagged queries. Integrated with DeepEye-SQL using Qwen3-Coder, ModularSQL preserves Set-EX at 72.06% while improving Multiset-EX from 65.86% to 67.75% (+1.89 pp). It flags 77 high-risk anomalies, while adding only 0.0076 in total LLM cost and 120 ms amortized latency per query. Cross-pipeline evaluation shows that the candidate-free detector and deterministic patches also transfer to independently released prediction sets. Overall, these results show that benchmark accuracy does not necessarily imply execution-safe SQL, and that lightweight, multiplicity-aware runtime guardrails can narrow this gap with modest computational overhead.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 7 of its 15 references that are in the map. First posted on arXiv 26 Aug 2026.
Builds on
7 of 15 references.
References · 7
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
- 2023Teaching Large Language Models to Self-Debugcited by 988
- 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
- 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
- 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
- 2025DeepEye-SQL: A Software-Engineering-Inspired Text-to-SQL Frameworkcited by 28
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026ACTS-SQL: Agentic and Critic-Oriented Tree-Structured SQL Correctness with Large Language Models
- 2025Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agents
- 2024BEAVER: An Enterprise Benchmark for Text-to-SQL
- 2024SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent Collaboration
- 2026PExA: Parallel Exploration Agent for Complex Text-to-SQL
- 2025SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications
- 2025RetrySQL: text-to-SQL training with retry data for self-correcting query generation
- 2026LIMIT: Less Is More for Instruction Tuning in Text-to-SQL