mlsanityData to 14 Oct 2026

SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation

arXiv 2604.06736 ↗First posted 8 Apr 2026cs.CL cs.DBcited by 1

Not on the map. Why?

Abstract (arXiv)

In Text-to-SQL tasks, large language models can generate structurally different SQL queries that return correct answers for the same intent. We call this phenomenon execution-correct structural divergence (ECSD). We introduce SQLStructEval, a framework that analyzes this behavior through canonical abstract syntax tree representations. We quantify structural diversity and agreement across generations. Experiments with different LLMs on multiple Text-to-SQL datasets, including Spider, document ECSD across models and datasets. Furthermore, our experiments demonstrate that generated queries are sensitive to question paraphrases and schema presentation. To address ECSD, we adopt a pipeline that first generates structured intermediate representations and then deterministically compiles them into SQL, improving execution accuracy and structural agreement among correct outputs. Structural analysis thus provides an additional diagnostic perspective that complements execution-based evaluation. Code is available at https://xanderzhou2022.github.io/AACL2026-SQLSTRUCTEVAL/.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 8 Apr 2026.

Builds on

36 of 51 references.

References · 36

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  8. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  9. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  10. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 424
  11. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 411
  12. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
24 more
  1. 2023How Language Model Hallucinations Can Snowballcited by 311
  2. 2020Unsupervised Translation of Programming Languagescited by 263
  3. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
  4. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
  5. 2019Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representationcited by 210
  6. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 197
  7. 2019SPoC: Search-based Pseudocode to Codecited by 176
  8. 2024CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQLcited by 138
  9. 2018Improving Text-to-SQL Evaluation Methodologycited by 135
  10. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 126
  11. 2021Towards Robustness of Text-to-SQL Models against Synonym Substitutioncited by 110
  12. 2025SQL-R1: Training Natural Language to SQL Reasoning Model By Reinforcement Learningcited by 68
  13. 2024Synthesizing Text-to-SQL Data from Weak and Strong LLMscited by 54
  14. 2023Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustnesscited by 53
  15. 2022Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expectcited by 47
  16. 2022Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbationcited by 28
  17. 2026FinCARDS: Card-Based Analyst Reranking for Financial Document Question Answeringcited by 16
  18. 2018Axiomatic Foundations and Algorithms for Deciding Semantic Equivalences of SQL Queriescited by 11
  19. 2024FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmarkcited by 11
  20. 2024PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQLcited by 7
  21. 2025JOLT-SQL: Joint Loss Tuning of Text-to-SQL with Confusion-aware Noisy Schema Samplingcited by 4
  22. 2025Dynamic Stability of LLM-Generated Codecited by 4
  23. 2025Measuring LLM Code Generation Stability via Structural Entropycited by 2
  24. 2025Instance-level Randomization: Toward More Stable LLM Evaluationscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Bounded Semantic Planning and Deterministic Compilation for Reliable Enterprise Text-to-SQLcited by 0