# Data Generation for Testing and Grading SQL Queries

arXiv: https://arxiv.org/abs/1411.6704 · HTML: https://mlsanity.com/paper/1411.6704/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2014.

## Built on by (most cited first, 7 of 7)

- [Real-time Workload Pattern Analysis for Large-scale Cloud Databases](https://mlsanity.com/paper/2307.02626/) (2023)
- [SpotIt: Evaluating Text-to-SQL Evaluation with Formal Verification](https://mlsanity.com/paper/2510.26840/) (2025)
- [Can the Rookies Cut the Tough Cookie? Exploring the Use of LLMs for SQL Equivalence Checking](https://mlsanity.com/paper/2412.05561/) (2024)
- [High-Fidelity And Complex Test Data Generation For Google SQL Code Generation Services](https://mlsanity.com/paper/2504.17203/) (2025)
- [EXPO-SQL: Execution-based Clause-level Policy Optimization for Text-to-SQL](https://mlsanity.com/paper/2606.23693/) (2026)
- [Performance Evaluation of Query Plan Recommendation with Apache Hadoop and Apache Spark](https://mlsanity.com/paper/2210.07143/) (2022)
- [Polygon: Symbolic Reasoning for SQL using Conflict-Driven Under-Approximation Search](https://mlsanity.com/paper/2504.06542/) (2025)

## Abstract (arXiv)

Correctness of SQL queries is usually tested by executing the queries on one or more datasets. Erroneous queries are often the results of small changes, or mutations of the correct query. A mutation Q' of a query Q is killed by a dataset D if Q(D) neq Q'(D). Earlier work on the XData system showed how to generate datasets that kill all mutations in a class of mutations that included join type and comparison operation mutations. In this paper, we extend the XData data generation techniques to handle a wider variety of SQL queries and a much larger class of mutations. We have also built a system for grading SQL queries using the datasets generated by XData. We present a study of the effectiveness of the datasets generated by the extended XData approach, using a variety of queries including queries submitted by students as part of a database course. We show that the XData datasets outperform predefined datasets as well as manual grading done earlier by teaching assistants, while also avoiding the drudgery of manual correction. Thus, we believe that our techniques will be of great value to database course instructors and TAs, particularly to those of MOOCs. It will also be valuable to database application developers and testers for testing SQL queries.