# Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation

arXiv: https://arxiv.org/abs/2212.09994 · HTML: https://mlsanity.com/paper/2212.09994/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 62 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Builds on (most cited first, 30 of 41)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://mlsanity.com/paper/1911.04942/) (2019)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases](https://mlsanity.com/paper/1909.05378/) (2019)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Few-NERD: A Few-Shot Named Entity Recognition Dataset](https://mlsanity.com/paper/2105.07464/) (2021)
- [SParC: Cross-Domain Semantic Parsing in Context](https://mlsanity.com/paper/1906.02285/) (2019)
- [Counter-fitting Word Vectors to Linguistic Constraints](https://mlsanity.com/paper/1603.00892/) (2016)
- [Structure-Grounded Pretraining for Text-to-SQL](https://mlsanity.com/paper/2010.12773/) (2020)
- [Towards Robustness of Text-to-SQL Models against Synonym Substitution](https://mlsanity.com/paper/2106.01065/) (2021)
- [A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization](https://mlsanity.com/paper/1902.01069/) (2019)
- [Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions](https://mlsanity.com/paper/1909.00786/) (2019)
- [Open Domain Question Answering over Tables via Dense Retrieval](https://mlsanity.com/paper/2103.12011/) (2021)
- [On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries](https://mlsanity.com/paper/2010.11246/) (2020)

## Built on by (most cited first, 28 of 28)

- [Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL](https://mlsanity.com/paper/2406.08426/) (2024)
- [A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability](https://mlsanity.com/paper/2303.13547/) (2023)
- [SCM: Enhancing Large Language Model with Self-Controlled Memory Framework](https://mlsanity.com/paper/2304.13343/) (2023)
- [A Survey on Employing Large Language Models for Text-to-SQL Tasks](https://mlsanity.com/paper/2407.15186/) (2024)
- [Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness](https://mlsanity.com/paper/2301.08881/) (2023)
- [RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations](https://mlsanity.com/paper/2306.14321/) (2023)
- [On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex](https://mlsanity.com/paper/2301.12868/) (2023)
- [CABINET: Content Relevance based Noise Reduction for Table Question Answering](https://mlsanity.com/paper/2402.01155/) (2024)
- [MultiSpider: Towards Benchmarking Multilingual Text-to-SQL Semantic Parsing](https://mlsanity.com/paper/2212.13492/) (2022)
- [TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring](https://mlsanity.com/paper/2403.15879/) (2024)
- [From Natural Language to SQL: Review of LLM-based Text-to-SQL Systems](https://mlsanity.com/paper/2410.01066/) (2024)
- [Error Detection for Text-to-SQL Semantic Parsing](https://mlsanity.com/paper/2305.13683/) (2023)
- [Solid-SQL: Enhanced Schema-linking based In-context Learning for Robust Text-to-SQL](https://mlsanity.com/paper/2412.12522/) (2024)
- [Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges](https://mlsanity.com/paper/2508.00217/) (2025)
- [SynTQA: Synergistic Table-based Question Answering via Mixture of Text-to-SQL and E2E TQA](https://mlsanity.com/paper/2409.16682/) (2024)
- [LOGOS: LLM-driven End-to-End Grounded Theory Development and Schema Induction for Qualitative Research](https://mlsanity.com/paper/2509.24294/) (2025)
- [EvoSchema: Towards Text-to-SQL Robustness Against Schema Evolution](https://mlsanity.com/paper/2603.10697/) (2026)
- [Improving Generalization in Semantic Parsing by Increasing Natural Language Variation](https://mlsanity.com/paper/2402.08666/) (2024)
- [TinySQL: A Progressive Text-to-SQL Dataset for Mechanistic Interpretability Research](https://mlsanity.com/paper/2503.12730/) (2025)
- [On the Structural Generalization in Text-to-SQL](https://mlsanity.com/paper/2301.04790/) (2023)
- [Squrve: A Unified and Modular Framework for Complex Real-World Text-to-SQL Tasks](https://mlsanity.com/paper/2510.24102/) (2025)
- [SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps](https://mlsanity.com/paper/2510.27532/) (2025)
- [SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation](https://mlsanity.com/paper/2604.06736/) (2026)
- [Comprehensive Solution Program Centric Pretraining for Table-and-Text Hybrid Numerical Reasoning](https://mlsanity.com/paper/2305.07475/) (2023)
- [Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQL](https://mlsanity.com/paper/2607.03833/) (2026)
- [ESQ-Bench: A Multi-Tier Enterprise Oracle Benchmark for Evaluating NL2SQL Dialect Generalization and Silent Semantic Divergence](https://mlsanity.com/paper/2608.23569/) (2026)
- [TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis](https://mlsanity.com/paper/2608.24145/) (2026)
- [DRL: A Deterministic Relational Middleware Layer for Transaction-Safe Enterprise NL2SQL Under Schema-Graph Scaling](https://mlsanity.com/paper/2608.26172/) (2026)

## Abstract (arXiv)

The robustness of Text-to-SQL parsers against adversarial perturbations plays a crucial role in delivering highly reliable applications. Previous studies along this line primarily focused on perturbations in the natural language question side, neglecting the variability of tables. Motivated by this, we propose the Adversarial Table Perturbation (ATP) as a new attacking paradigm to measure the robustness of Text-to-SQL models. Following this proposition, we curate ADVETA, the first robustness evaluation benchmark featuring natural and realistic ATPs. All tested state-of-the-art models experience dramatic performance drops on ADVETA, revealing models' vulnerability in real-world practices. To defend against ATP, we build a systematic adversarial training example generation framework tailored for better contextualization of tabular data. Experiments show that our approach not only brings the best robustness improvement against table-side perturbations but also substantially empowers models against NL-side perturbations. We release our benchmark and code at: https://github.com/microsoft/ContextualSP.