mlsanityData to 14 Oct 2026

Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL Translation

arXiv 2402.08100 ↗First posted 12 Feb 2024cs.CL cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Understanding textual description to generate code seems to be an achieved capability of instruction-following Large Language Models (LLMs) in zero-shot scenario. However, there is a severe possibility that this translation ability may be influenced by having seen target textual descriptions and the related code. This effect is known as Data Contamination. In this study, we investigate the impact of Data Contamination on the performance of GPT-3.5 in the Text-to-SQL code-generating tasks. Hence, we introduce a novel method to detect Data Contamination in GPTs and examine GPT-3.5's Text-to-SQL performances using the known Spider Dataset and our new unfamiliar dataset Termite. Furthermore, we analyze GPT-3.5's efficacy on databases with modified information via an adversarial table disconnection (ATD) approach, complicating Text-to-SQL tasks by removing structural pieces of information from the database. Our results indicate a significant performance drop in GPT-3.5 on the unfamiliar Termite dataset, even with ATD modifications, highlighting the effect of Data Contamination on LLMs in Text-to-SQL translation tasks.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 16 of its 27 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

16 of 27 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Extracting Training Data from Large Language Modelscited by 2,189
  3. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  5. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  6. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  7. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
  8. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  9. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  10. 2017SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learningcited by 135
  11. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 126
  12. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 111
4 more
  1. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  2. 2023ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-Thoughtcited by 65
  3. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 50
  4. 2022The Dark Side of the Language: Pre-trained Transformers in the DarkNetcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 13; 2026: 1.

21 papers.

Sort:

By area:

  1. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  2. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 65
  3. 2024MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmarkcited by 41
  4. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  5. 2025A Survey on Data Contamination for Large Language Modelscited by 35
  6. 2024Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediationcited by 30
  7. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  8. 2024A Comprehensive Survey of Contamination Detection Methods in Large Language Modelscited by 12
  9. 2025VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contaminationcited by 12
  10. 2024FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmarkcited by 11
  11. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
  12. 2025On The Fragility of Benchmark Contamination Detection in Reasoning Modelscited by 9
9 more
  1. 2025Private Memorization Editing: Turning Memorization into a Defense to Strengthen Data Privacy in Large Language Modelscited by 7
  2. 2025Mining for Species, Locations, Habitats, and Ecosystems from Scientific Papers in Invasion Biology: A Large-Scale Exploratory Study with Large Language Modelscited by 4
  3. 2025Generative Induction of Dialogue Task Schemas with Streaming Refinement and Simulated Interactionscited by 2
  4. 2025Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?cited by 2
  5. 2025Protoknowledge Shapes Behaviour of LLMs in Downstream Tasks: Memorization and Generalization with Knowledge Graphscited by 1
  6. 2025Beyond Frequency: The Role of Redundancy in Large Language Model Memorizationcited by 0
  7. 2025DynaQuery: A Self-Adapting Framework for Querying Structured and Multimodal Datacited by 0
  8. 2025DCMM-SQL: Automated Data-Centric Pipeline and Multi-Model Collaboration Training for Text-to-SQL Modelcited by 0
  9. 2026SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarkscited by 0