# Don't paraphrase, detect! Rapid and Effective Data Collection for Semantic Parsing

arXiv: https://arxiv.org/abs/1908.09940 · HTML: https://mlsanity.com/paper/1908.09940/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Aug 2019.

## Built on by (most cited first, 24 of 24)

- [Diverse Demonstrations Improve In-context Compositional Generalization](https://mlsanity.com/paper/2212.06800/) (2022)
- [Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing](https://mlsanity.com/paper/2205.12253/) (2022)
- [Improving Compositional Generalization with Latent Structure and Data Augmentation](https://mlsanity.com/paper/2112.07610/) (2021)
- [Learning to Synthesize Data for Semantic Parsing](https://mlsanity.com/paper/2104.05827/) (2021)
- [Data Augmentation with Hierarchical SQL-to-Question Generation for Cross-domain Text-to-SQL Parsing](https://mlsanity.com/paper/2103.02227/) (2021)
- [On the Evaluation Metrics for Paraphrase Generation](https://mlsanity.com/paper/2202.08479/) (2022)
- [Towards Knowledge-Intensive Text-to-SQL Semantic Parsing with Formulaic Knowledge](https://mlsanity.com/paper/2301.01067/) (2023)
- [From Paraphrasing to Semantic Parsing: Unsupervised Semantic Parsing via Synchronous Semantic Decoding](https://mlsanity.com/paper/2106.06228/) (2021)
- [On The Ingredients of an Effective Zero-shot Semantic Parser](https://mlsanity.com/paper/2110.08381/) (2021)
- [GraphQ IR: Unifying the Semantic Parsing of Graph Query Languages with One Intermediate Representation](https://mlsanity.com/paper/2205.12078/) (2022)
- [Expanding End-to-End Question Answering on Differentiable Knowledge Graphs with Intersection](https://mlsanity.com/paper/2109.05808/) (2021)
- [Generating Data for Symbolic Language with Large Language Models](https://mlsanity.com/paper/2305.13917/) (2023)
- [Low-Resource Task-Oriented Semantic Parsing via Intrinsic Modeling](https://mlsanity.com/paper/2104.07224/) (2021)
- [Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation](https://mlsanity.com/paper/2203.08445/) (2022)
- [Few-Shot Semantic Parsing for New Predicates](https://mlsanity.com/paper/2101.10708/) (2021)
- [Pre-training Universal Language Representation](https://mlsanity.com/paper/2105.14478/) (2021)
- [Compositional Generalization for Natural Language Interfaces to Web APIs](https://mlsanity.com/paper/2112.05209/) (2021)
- [Paraphrasing Techniques for Maritime QA system](https://mlsanity.com/paper/2203.10854/) (2022)
- [Diverse Parallel Data Synthesis for Cross-Database Adaptation of Text-to-SQL Parsers](https://mlsanity.com/paper/2210.16613/) (2022)
- [Improving Cross-Lingual Transfer through Subtree-Aware Word Reordering](https://mlsanity.com/paper/2310.13583/) (2023)
- [Semantic-aware Contrastive Learning for More Accurate Semantic Parsing](https://mlsanity.com/paper/2301.07919/) (2023)
- [On graph-based reentrancy-free semantic parsing](https://mlsanity.com/paper/2302.07679/) (2023)
- [Reranking for Natural Language Generation from Logical Forms: A Study based on Large Language Models](https://mlsanity.com/paper/2309.12294/) (2023)
- [Semantic Parsing in Limited Resource Conditions](https://mlsanity.com/paper/2309.07429/) (2023)

## Abstract (arXiv)

A major hurdle on the road to conversational interfaces is the difficulty in collecting data that maps language utterances to logical forms. One prominent approach for data collection has been to automatically generate pseudo-language paired with logical forms, and paraphrase the pseudo-language to natural language through crowdsourcing (Wang et al., 2015). However, this data collection procedure often leads to low performance on real data, due to a mismatch between the true distribution of examples and the distribution induced by the data collection procedure. In this paper, we thoroughly analyze two sources of mismatch in this process: the mismatch in logical form distribution and the mismatch in language distribution between the true and induced distributions. We quantify the effects of these mismatches, and propose a new data collection approach that mitigates them. Assuming access to unlabeled utterances from the true distribution, we combine crowdsourcing with a paraphrase model to detect correct logical forms for the unlabeled utterances. On two datasets, our method leads to 70.6 accuracy on average on the true distribution, compared to 51.3 in paraphrasing-based data collection.