# Towards Robustness of Text-to-SQL Models against Synonym Substitution

arXiv: https://arxiv.org/abs/2106.01065 · HTML: https://mlsanity.com/paper/2106.01065/

## Key facts

- Cited by 110 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 20 of 20)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://mlsanity.com/paper/1911.04942/) (2019)
- [Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation](https://mlsanity.com/paper/1905.08205/) (2019)
- [Counter-fitting Word Vectors to Linguistic Constraints](https://mlsanity.com/paper/1603.00892/) (2016)
- [Learning a Neural Semantic Parser from User Feedback](https://mlsanity.com/paper/1704.08760/) (2017)
- [Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions](https://mlsanity.com/paper/1909.00786/) (2019)
- [SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Task](https://mlsanity.com/paper/1810.05237/) (2018)
- [Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing](https://mlsanity.com/paper/1905.06241/) (2019)
- [Generating Fluent Adversarial Examples for Natural Languages](https://mlsanity.com/paper/2007.06174/) (2020)
- [Global Reasoning over Database Structures for Text-to-SQL Parsing](https://mlsanity.com/paper/1908.11214/) (2019)
- [Photon: A Robust Cross-Domain Text-to-SQL System](https://mlsanity.com/paper/2007.15280/) (2020)
- [SpatialNLI: A Spatial Domain Natural Language Interface to Databases Using Spatial Comprehension](https://mlsanity.com/paper/1908.10917/) (2019)
- [Generating Semantically Valid Adversarial Questions for TableQA](https://mlsanity.com/paper/2005.12696/) (2020)
- [Transferable Natural Language Interface to Structured Queries aided by Adversarial Generation](https://mlsanity.com/paper/1812.01245/) (2018)

## Built on by (most cited first, 30 of 110)

- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [StructGPT: A General Framework for Large Language Model to Reason over Structured Data](https://mlsanity.com/paper/2305.09645/) (2023)
- [RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL](https://mlsanity.com/paper/2302.05965/) (2023)
- [CodeS: Towards Building Open-source Language Models for Text-to-SQL](https://mlsanity.com/paper/2402.16347/) (2024)
- [Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL](https://mlsanity.com/paper/2406.08426/) (2024)
- [A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability](https://mlsanity.com/paper/2303.13547/) (2023)
- [VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use](https://mlsanity.com/paper/2509.01055/) (2025)
- [Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsing](https://mlsanity.com/paper/2301.07507/) (2023)
- [Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization](https://mlsanity.com/paper/2109.05157/) (2021)
- [OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale](https://mlsanity.com/paper/2503.02240/) (2025)
- [Natural Language to Code Generation in Interactive Data Science Notebooks](https://mlsanity.com/paper/2212.09248/) (2022)
- [A Survey on Employing Large Language Models for Text-to-SQL Tasks](https://mlsanity.com/paper/2407.15186/) (2024)
- [ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-Thought](https://mlsanity.com/paper/2310.17342/) (2023)
- [Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQL](https://mlsanity.com/paper/2503.23157/) (2025)
- [A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?](https://mlsanity.com/paper/2408.05109/) (2024)
- [Synthesizing Text-to-SQL Data from Weak and Strong LLMs](https://mlsanity.com/paper/2408.03256/) (2024)
- [Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness](https://mlsanity.com/paper/2301.08881/) (2023)
- [A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions](https://mlsanity.com/paper/2208.13629/) (2022)
- [Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expect](https://mlsanity.com/paper/2208.10099/) (2022)
- [Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies](https://mlsanity.com/paper/2305.12586/) (2023)
- [ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems](https://mlsanity.com/paper/2306.04743/) (2023)
- [DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models](https://mlsanity.com/paper/2402.01117/) (2024)
- [RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations](https://mlsanity.com/paper/2306.14321/) (2023)
- [SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended)](https://mlsanity.com/paper/2306.00739/) (2023)
- [S2SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parsers](https://mlsanity.com/paper/2203.06958/) (2022)
- [Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark](https://mlsanity.com/paper/2402.12243/) (2024)
- [PURPLE: Making a Large Language Model a Better SQL Writer](https://mlsanity.com/paper/2403.20014/) (2024)
- [Evaluating Cross-Domain Text-to-SQL Models and Benchmarks](https://mlsanity.com/paper/2310.18538/) (2023)
- [SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Models](https://mlsanity.com/paper/2403.02951/) (2024)
- [Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsing](https://mlsanity.com/paper/2206.14017/) (2022)

## Abstract (arXiv)

Recently, there has been significant progress in studying neural networks to translate text descriptions into SQL queries. Despite achieving good performance on some public benchmarks, existing text-to-SQL models typically rely on the lexical matching between words in natural language (NL) questions and tokens in table schemas, which may render the models vulnerable to attacks that break the schema linking mechanism. In this work, we investigate the robustness of text-to-SQL models to synonym substitution. In particular, we introduce Spider-Syn, a human-curated dataset based on the Spider benchmark for text-to-SQL translation. NL questions in Spider-Syn are modified from Spider, by replacing their schema-related words with manually selected synonyms that reflect real-world question paraphrases. We observe that the accuracy dramatically drops by eliminating such explicit correspondence between NL questions and table schemas, even if the synonyms are not adversarially selected to conduct worst-case adversarial attacks. Finally, we present two categories of approaches to improve the model robustness. The first category of approaches utilizes additional synonym annotations for table schemas by modifying the model input, while the second category is based on adversarial training. We demonstrate that both categories of approaches significantly outperform their counterparts without the defense, and the first category of approaches are more effective.