# On Robustness of Neural Semantic Parsers

arXiv: https://arxiv.org/abs/2102.01563 · HTML: https://mlsanity.com/paper/2102.01563/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 38 references that are in the map.
- First posted on arXiv 2 Feb 2021.

## Builds on (most cited first, 30 of 33)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://mlsanity.com/paper/1911.04942/) (2019)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [Simple Black-box Adversarial Attacks](https://mlsanity.com/paper/1905.07121/) (2019)
- [Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation](https://mlsanity.com/paper/1905.08205/) (2019)
- [Language to Logical Form with Neural Attention](https://mlsanity.com/paper/1601.01280/) (2016)
- [Data Recombination for Neural Semantic Parsing](https://mlsanity.com/paper/1606.03622/) (2016)
- [Coarse-to-Fine Decoding for Neural Semantic Parsing](https://mlsanity.com/paper/1805.04793/) (2018)
- [Crafting Adversarial Input Sequences for Recurrent Neural Networks](https://mlsanity.com/paper/1604.08275/) (2016)
- [Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples](https://mlsanity.com/paper/1803.01128/) (2018)
- [TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation](https://mlsanity.com/paper/1810.02720/) (2018)
- [On Adversarial Examples for Character-Level Neural Machine Translation](https://mlsanity.com/paper/1806.09030/) (2018)
- [A Survey on Semantic Parsing](https://mlsanity.com/paper/1812.00978/) (2018)
- [On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models](https://mlsanity.com/paper/1903.06620/) (2019)
- [Neural Syntactic Preordering for Controlled Paraphrase Generation](https://mlsanity.com/paper/2005.02013/) (2020)
- [Corpora Generation for Grammatical Error Correction](https://mlsanity.com/paper/1904.05780/) (2019)
- [On the Robustness of the CVPR 2018 White-Box Adversarial Example Defenses](https://mlsanity.com/paper/1804.03286/) (2018)
- [Natural Language to Structured Query Generation via Meta-Learning](https://mlsanity.com/paper/1803.02400/) (2018)
- [Question Generation from SQL Queries Improves Neural Semantic Parsing](https://mlsanity.com/paper/1808.06304/) (2018)
- [Sequence-to-Action: End-to-End Semantic Graph Generation for Semantic Parsing](https://mlsanity.com/paper/1809.00773/) (2018)
- [Comparing Attention-based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehension](https://mlsanity.com/paper/1808.08744/) (2018)
- [HotFlip: White-Box Adversarial Examples for Text Classification](https://mlsanity.com/paper/1712.06751/) (2017)

## Built on by (most cited first, 14 of 14)

- [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021)
- [Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness](https://mlsanity.com/paper/2301.08881/) (2023)
- [FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing](https://mlsanity.com/paper/2305.17497/) (2023)
- [On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex](https://mlsanity.com/paper/2301.12868/) (2023)
- [Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation](https://mlsanity.com/paper/2212.09994/) (2022)
- [Prompt Perturbation Consistency Learning for Robust Language Models](https://mlsanity.com/paper/2402.15833/) (2024)
- [Energy-Latency Attacks to On-Device Neural Networks via Sponge Poisoning](https://mlsanity.com/paper/2305.03888/) (2023)
- [Improving Generalization in Semantic Parsing by Increasing Natural Language Variation](https://mlsanity.com/paper/2402.08666/) (2024)
- [The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning](https://mlsanity.com/paper/2305.12737/) (2023)
- [Enhancing Robustness of AI Offensive Code Generators via Data Augmentation](https://mlsanity.com/paper/2306.05079/) (2023)
- [Can NMT Understand Me? Towards Perturbation-based Evaluation of NMT Models for Code Generation](https://mlsanity.com/paper/2203.15319/) (2022)
- [More Robust Schema-Guided Dialogue State Tracking via Tree-Based Paraphrase Ranking](https://mlsanity.com/paper/2303.09905/) (2023)
- [Beyond the Model: Data Pre-processing Attack to Deep Learning Models in Android Apps](https://mlsanity.com/paper/2305.03963/) (2023)
- [Grounding Description-Driven Dialogue State Trackers with Knowledge-Seeking Turns](https://mlsanity.com/paper/2309.13448/) (2023)

## Abstract (arXiv)

Semantic parsing maps natural language (NL) utterances into logical forms (LFs), which underpins many advanced NLP problems. Semantic parsers gain performance boosts with deep neural networks, but inherit vulnerabilities against adversarial examples. In this paper, we provide the empirical study on the robustness of semantic parsers in the presence of adversarial attacks. Formally, adversaries of semantic parsing are considered to be the perturbed utterance-LF pairs, whose utterances have exactly the same meanings as the original ones. A scalable methodology is proposed to construct robustness test sets based on existing benchmark corpora. Our results answered five research questions in measuring the sate-of-the-art parsers' performance on robustness test sets, and evaluating the effect of data augmentation.