# Unnatural Language Processing: Bridging the Gap Between Synthetic and Natural Language Data

arXiv: https://arxiv.org/abs/2004.13645 · HTML: https://mlsanity.com/paper/2004.13645/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2020.

## Built on by (most cited first, 16 of 16)

- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning](https://mlsanity.com/paper/2107.02794/) (2021)
- ["No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy](https://mlsanity.com/paper/2301.02555/) (2023)
- [Generate, Annotate, and Learn: NLP with Synthetic Text](https://mlsanity.com/paper/2106.06168/) (2021)
- [SeqZero: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models](https://mlsanity.com/paper/2205.07381/) (2022)
- [Communicating Natural Programs to Humans and Machines](https://mlsanity.com/paper/2106.07824/) (2021)
- [LILA: Language-Informed Latent Actions](https://mlsanity.com/paper/2111.03205/) (2021)
- [The Power of Prompt Tuning for Low-Resource Semantic Parsing](https://mlsanity.com/paper/2110.08525/) (2021)
- [From Paraphrasing to Semantic Parsing: Unsupervised Semantic Parsing via Synchronous Semantic Decoding](https://mlsanity.com/paper/2106.06228/) (2021)
- [On The Ingredients of an Effective Zero-shot Semantic Parser](https://mlsanity.com/paper/2110.08381/) (2021)
- [Synthetic Data in Healthcare](https://mlsanity.com/paper/2304.03243/) (2023)
- [Grounding Spatio-Temporal Language with Transformers](https://mlsanity.com/paper/2106.08858/) (2021)
- [SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Data](https://mlsanity.com/paper/2210.02989/) (2022)
- [Semantic-aware Contrastive Learning for More Accurate Semantic Parsing](https://mlsanity.com/paper/2301.07919/) (2023)
- [Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning](https://mlsanity.com/paper/2107.09285/) (2021)

## Abstract (arXiv)

Large, human-annotated datasets are central to the development of natural language processing models. Collecting these datasets can be the most challenging part of the development process. We address this problem by introducing a general purpose technique for ``simulation-to-real'' transfer in language understanding problems with a delimited set of target behaviors, making it possible to develop models that can interpret natural utterances without natural training data. We begin with a synthetic data generation procedure, and train a model that can accurately interpret utterances produced by the data generator. To generalize to natural utterances, we automatically find projections of natural language utterances onto the support of the synthetic language, using learned sentence embeddings to define a distance metric. With only synthetic training data, our approach matches or outperforms state-of-the-art models trained on natural language data in several domains. These results suggest that simulation-to-real transfer is a practical framework for developing NLP applications, and that improved models for transfer might provide wide-ranging improvements in downstream tasks.