mlsanityData to 7 Oct 2026

Towards Robustness of Text-to-SQL Models against Synonym Substitution

arXiv 2106.01065 ↗First posted 2 Jun 2021cs.CLcited by 110

Not on the map. Why?

Abstract (arXiv)

Recently, there has been significant progress in studying neural networks to translate text descriptions into SQL queries. Despite achieving good performance on some public benchmarks, existing text-to-SQL models typically rely on the lexical matching between words in natural language (NL) questions and tokens in table schemas, which may render the models vulnerable to attacks that break the schema linking mechanism. In this work, we investigate the robustness of text-to-SQL models to synonym substitution. In particular, we introduce Spider-Syn, a human-curated dataset based on the Spider benchmark for text-to-SQL translation. NL questions in Spider-Syn are modified from Spider, by replacing their schema-related words with manually selected synonyms that reflect real-world question paraphrases. We observe that the accuracy dramatically drops by eliminating such explicit correspondence between NL questions and table schemas, even if the synonyms are not adversarially selected to conduct worst-case adversarial attacks. Finally, we present two categories of approaches to improve the model robustness. The first category of approaches utilizes additional synonym annotations for table schemas by modifying the model input, while the second category is based on adversarial training. We demonstrate that both categories of approaches significantly outperform their counterparts without the defense, and the first category of approaches are more effective.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Builds on 20 of its 35 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

20 of 35 references.

References · 20

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  2. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  3. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  4. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  5. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  6. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  7. 2018Generating Natural Language Adversarial Examplescited by 422
  8. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 420
  9. 2019Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representationcited by 210
  10. 2016Counter-fitting Word Vectors to Linguistic Constraintscited by 138
  11. 2017Learning a Neural Semantic Parser from User Feedbackcited by 98
  12. 2019Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questionscited by 80
8 more
  1. 2018SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Taskcited by 74
  2. 2019Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsingcited by 74
  3. 2020Generating Fluent Adversarial Examples for Natural Languagescited by 49
  4. 2019Global Reasoning over Database Structures for Text-to-SQL Parsingcited by 43
  5. 2020Photon: A Robust Cross-Domain Text-to-SQL Systemcited by 26
  6. 2019SpatialNLI: A Spatial Domain Natural Language Interface to Databases Using Spatial Comprehensioncited by 5
  7. 2020Generating Semantically Valid Adversarial Questions for TableQAcited by 4
  8. 2018Transferable Natural Language Interface to Structured Queries aided by Adversarial Generationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 13; 2023: 25; 2024: 23; 2025: 31; 2026: 17.

110 papers.

Sort:

By area:

  1. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  2. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 339
  3. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 191
  4. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  5. 2023A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capabilitycited by 118
  6. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  7. 2023Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsingcited by 93
  8. 2021Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalizationcited by 92
  9. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  10. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 90
  11. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  12. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
38 more
  1. 2023ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-Thoughtcited by 65
  2. 2025Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQLcited by 59
  3. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 54
  4. 2024Synthesizing Text-to-SQL Data from Weak and Strong LLMscited by 53
  5. 2022A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directionscited by 51
  6. 2023Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustnesscited by 50
  7. 2022Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expectcited by 47
  8. 2024DTS-SQL: Decomposed Text-to-SQL with Small Large Language Modelscited by 45
  9. 2023Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategiescited by 44
  10. 2023SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended)cited by 42
  11. 2023ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systemscited by 42
  12. 2022S2SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parserscited by 41
  13. 2023RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbationscited by 41
  14. 2024PURPLE: Making a Large Language Model a Better SQL Writercited by 34
  15. 2024Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmarkcited by 33
  16. 2023Evaluating Cross-Domain Text-to-SQL Models and Benchmarkscited by 32
  17. 2024SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Modelscited by 32
  18. 2022Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsingcited by 30
  19. 2024A Survey of Table Reasoning with Large Language Modelscited by 29
  20. 2022Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbationcited by 28
  21. 2023Natural Language Interfaces for Tabular Data Querying and Visualization: A Surveycited by 27
  22. 2025Reward-SQL: Boosting Text-to-SQL via Stepwise Execution-Aware Reasoning and Process-Supervised Rewardscited by 25
  23. 2022Semantic Enhanced Text-to-SQL Parsing via Iteratively Learning Schema Linking Graphcited by 22
  24. 2022MultiSpider: Towards Benchmarking Multilingual Text-to-SQL Semantic Parsingcited by 22
  25. 2025DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AIcited by 22
  26. 2023Text-to-SQL Error Correction with Language Models of Codecited by 21
  27. 2023Divide and Prompt: Chain of Thought Prompting for Text-to-SQLcited by 20
  28. 2023Interactive Text-to-SQL Generation via Editable Step-by-Step Explanationscited by 19
  29. 2023Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrievalcited by 16
  30. 2024Improving Retrieval-augmented Text-to-SQL with AST-based Ranking and Schema Pruningcited by 16
  31. 2023UNITE: A Unified Benchmark for Text-to-SQL Evaluationcited by 14
  32. 2025Think2SQL: Reinforce LLM Reasoning Capabilities for Text2SQLcited by 14
  33. 2022UniRPG: Unified Discrete Reasoning over Table and Text as Program Generationcited by 13
  34. 2024ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQLcited by 13
  35. 2023Retrieval-augmented GPT-3.5-based Text-to-SQL Framework with Sample-aware Prompting and Dynamic Revision Chaincited by 12
  36. 2025Disambiguate First, Parse Later: Generating Interpretations for Ambiguity Resolution in Semantic Parsingcited by 12
  37. 2024Solid-SQL: Enhanced Schema-linking based In-context Learning for Robust Text-to-SQLcited by 11
  38. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11

50 most cited of 110 All 110 (JSON)