# Do LLMs Align with My Task? Evaluating Text-to-SQL via Dataset Alignment

arXiv: https://arxiv.org/abs/2510.04919 · HTML: https://mlsanity.com/paper/2510.04919/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 29 references that are in the map.
- First posted on arXiv 6 Oct 2025.

## Builds on (most cited first, 24 of 24)

- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://mlsanity.com/paper/2304.11015/) (2023)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [A Survey on Data Selection for Language Models](https://mlsanity.com/paper/2402.16827/) (2024)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [BOND: Aligning LLMs with Best-of-N Distillation](https://mlsanity.com/paper/2407.14622/) (2024)
- [Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2404.02657/) (2024)
- [Synthesizing Text-to-SQL Data from Weak and Strong LLMs](https://mlsanity.com/paper/2408.03256/) (2024)
- [DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models](https://mlsanity.com/paper/2402.01117/) (2024)
- [SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended)](https://mlsanity.com/paper/2306.00739/) (2023)
- [MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning](https://mlsanity.com/paper/2311.02303/) (2023)
- [Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs](https://mlsanity.com/paper/2405.02774/) (2024)
- [Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning](https://mlsanity.com/paper/2407.05040/) (2024)
- [SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging](https://mlsanity.com/paper/2408.12733/) (2024)
- [Question Answering through Transfer Learning from Large Fine-grained Supervision Data](https://mlsanity.com/paper/1702.02171/) (2017)
- [SQL-Encoder: Improving NL2SQL In-Context Learning Through a Context-Aware Encoder](https://mlsanity.com/paper/2403.16204/) (2024)

## Abstract (arXiv)

Supervised Fine-Tuning (SFT) is an effective method for adapting Large Language Models (LLMs) on downstream tasks. However, variability in training data can hinder a model's ability to generalize across domains. This paper studies the problem of dataset alignment for Natural Language to SQL (NL2SQL or text to SQL), examining how well SFT training data matches the structural characteristics of target queries and how this alignment impacts model performance. We hypothesize that alignment can be accurately estimated by comparing the distributions of structural SQL features across the training set, target data, and the model's predictions prior to SFT. Through comprehensive experiments on three large cross-domain NL2SQL benchmarks and multiple model families, we show that structural alignment is a strong predictor of fine-tuning success. When alignment is high, SFT yields substantial gains in accuracy and SQL generation quality; when alignment is low, improvements are marginal or absent. These findings highlight the importance of alignment-aware data selection for effective fine-tuning and generalization in NL2SQL tasks.