# High-Fidelity And Complex Test Data Generation For Google SQL Code Generation Services

arXiv: https://arxiv.org/abs/2504.17203 · HTML: https://mlsanity.com/paper/2504.17203/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 46 references that are in the map.
- First posted on arXiv 24 Apr 2025.

## Builds on (most cited first, 21 of 21)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Language Models are Realistic Tabular Data Generators](https://mlsanity.com/paper/2210.06280/) (2022)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space](https://mlsanity.com/paper/2310.09656/) (2023)
- [Can Foundation Models Wrangle Your Data?](https://mlsanity.com/paper/2205.09911/) (2022)
- [Machine Learning for Synthetic Data Generation: A Review](https://mlsanity.com/paper/2302.04062/) (2023)
- [Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study](https://mlsanity.com/paper/2305.13062/) (2023)
- [Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations](https://mlsanity.com/paper/2310.07849/) (2023)
- [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023)
- [CoDi: Co-evolving Contrastive Diffusion Models for Mixed-type Tabular Synthesis](https://mlsanity.com/paper/2304.12654/) (2023)
- [DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows](https://mlsanity.com/paper/2402.10379/) (2024)
- [Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing](https://mlsanity.com/paper/2212.04732/) (2022)
- [Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQL](https://mlsanity.com/paper/2501.12372/) (2025)
- [Schema Matching with Large Language Models: an Experimental Study](https://mlsanity.com/paper/2407.11852/) (2024)
- [Column Type Annotation using ChatGPT](https://mlsanity.com/paper/2306.00745/) (2023)
- [VeriEQL: Bounded Equivalence Verification for Complex SQL Queries with Integrity Constraints](https://mlsanity.com/paper/2403.03193/) (2024)
- [Data Generation for Testing and Grading SQL Queries](https://mlsanity.com/paper/1411.6704/) (2014)
- [Estimating the Impact of Unknown Unknowns on Aggregate Query Results](https://mlsanity.com/paper/1507.05591/) (2015)
- [Data Generation for Testing Complex Queries](https://mlsanity.com/paper/2409.18821/) (2024)
- [Can the Rookies Cut the Tough Cookie? Exploring the Use of LLMs for SQL Equivalence Checking](https://mlsanity.com/paper/2412.05561/) (2024)

## Built on by (most cited first, 1 of 1)

- [TabQueryBench: A Query-Centric Benchmark for Synthetic Tabular Data](https://mlsanity.com/paper/2607.03926/) (2026)

## Abstract (arXiv)

The demand for high-fidelity test data is paramount in industrial settings where access to production data is largely restricted. Traditional data generation methods often fall short, struggling with low-fidelity and the ability to model complex data structures and semantic relationships that are critical for testing complex SQL code generation services like Natural Language to SQL (NL2SQL). In this paper, we address the critical need for generating syntactically correct and semantically relevant high-fidelity mock data for complex data structures that includes columns with nested structures that we frequently encounter in Google workloads. We highlight the limitations of existing approaches used in production, particularly their inability to handle large and complex data structures, as well as the lack of semantically coherent test data that lead to limited test coverage. We demonstrate that by leveraging Large Language Models (LLMs) and incorporating strategic pre- and post-processing steps, we can generate syntactically correct and semantically relevant high-fidelity test data that adheres to complex structural constraints and maintains semantic integrity to the SQL test targets (queries/functions). This approach supports comprehensive testing of complex SQL queries involving joins, aggregations, and even deeply nested subqueries, ensuring robust evaluation of SQL code generation services, like NL2SQL and SQL Code Assistant. Our results demonstrate the practical utility of an LLM (textitGemini) based test data generation for industrial SQL code generation services where generating high-fidelity test data is essential due to the frequent unavailability and inaccessibility of production datasets for testing.