# Neural Data Augmentation via Example Extrapolation

arXiv: https://arxiv.org/abs/2102.01335 · HTML: https://mlsanity.com/paper/2102.01335/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 2 Feb 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://mlsanity.com/paper/2010.15980/) (2020)
- [Not Enough Data? Deep Learning to the Rescue!](https://mlsanity.com/paper/1911.03118/) (2019)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [Good-Enough Compositional Data Augmentation](https://mlsanity.com/paper/1904.09545/) (2019)
- [Zero-Shot Entity Linking by Reading Entity Descriptions](https://mlsanity.com/paper/1906.07348/) (2019)
- [Data Recombination for Neural Semantic Parsing](https://mlsanity.com/paper/1606.03622/) (2016)
- [FewRel 2.0: Towards More Challenging Few-Shot Relation Classification](https://mlsanity.com/paper/1910.07124/) (2019)
- [Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network](https://mlsanity.com/paper/2006.05702/) (2020)
- [Learning to Recombine and Resample Data for Compositional Generalization](https://mlsanity.com/paper/2010.03706/) (2020)
- [Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference](https://mlsanity.com/paper/2010.13009/) (2020)
- [A Closer Look At Feature Space Data Augmentation For Few-Shot Intent Classification](https://mlsanity.com/paper/1910.04176/) (2019)
- [Towards Zero-Shot Frame Semantic Parsing for Domain Scaling](https://mlsanity.com/paper/1707.02363/) (2017)
- [Learning to Classify Intents and Slot Labels Given a Handful of Examples](https://mlsanity.com/paper/2004.10793/) (2020)
- [Entity-Switched Datasets: An Approach to Auditing the In-Domain Robustness of Named Entity Recognition Models](https://mlsanity.com/paper/2004.04123/) (2020)
- [Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning](https://mlsanity.com/paper/2010.02405/) (2020)

## Built on by (most cited first, 30 of 54)

- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [RARR: Researching and Revising What Language Models Say, Using Language Models](https://mlsanity.com/paper/2210.08726/) (2022)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [Towards Zero-Label Language Learning](https://mlsanity.com/paper/2109.09193/) (2021)
- [Tailor: Generating and Perturbing Text with Semantic Controls](https://mlsanity.com/paper/2107.07150/) (2021)
- [ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback](https://mlsanity.com/paper/2210.12329/) (2022)
- [RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction](https://mlsanity.com/paper/2203.09101/) (2022)
- [Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness](https://mlsanity.com/paper/2301.08881/) (2023)
- [Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning](https://mlsanity.com/paper/2205.12679/) (2022)
- [Data Augmentation for Intent Classification with Off-the-shelf Large Language Models](https://mlsanity.com/paper/2204.01959/) (2022)
- [Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning](https://mlsanity.com/paper/2211.03044/) (2022)
- [Controllable Semantic Parsing via Retrieval Augmentation](https://mlsanity.com/paper/2110.08458/) (2021)
- [STraTA: Self-Training with Task Augmentation for Better Few-shot Learning](https://mlsanity.com/paper/2109.06270/) (2021)
- [LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Tagging](https://mlsanity.com/paper/2209.09900/) (2022)
- [Propagating Knowledge Updates to LMs Through Distillation](https://mlsanity.com/paper/2306.09306/) (2023)
- [ConvFiT: Conversational Fine-Tuning of Pretrained Language Models](https://mlsanity.com/paper/2109.10126/) (2021)
- [Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Information](https://mlsanity.com/paper/2302.05096/) (2023)
- [Few-shot Controllable Style Transfer for Low-Resource Multilingual Settings](https://mlsanity.com/paper/2110.07385/) (2021)
- [CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsing](https://mlsanity.com/paper/2210.07074/) (2022)
- [Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System](https://mlsanity.com/paper/2205.04006/) (2022)
- [SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created Through Human-Machine Collaboration](https://mlsanity.com/paper/2305.17696/) (2023)
- [SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation](https://mlsanity.com/paper/2405.10040/) (2024)
- [Multi-Vector Retrieval as Sparse Alignment](https://mlsanity.com/paper/2211.01267/) (2022)
- [Generative Conversational Networks](https://mlsanity.com/paper/2106.08484/) (2021)

## Abstract (arXiv)

In many applications of machine learning, certain categories of examples may be underrepresented in the training data, causing systems to underperform on such "few-shot" cases at test time. A common remedy is to perform data augmentation, such as by duplicating underrepresented examples, or heuristically synthesizing new examples. But these remedies often fail to cover the full diversity and complexity of real examples. We propose a data augmentation approach that performs neural Example Extrapolation (Ex2). Given a handful of exemplars sampled from some distribution, Ex2 synthesizes new examples that also belong to the same distribution. The Ex2 model is learned by simulating the example generation procedure on data-rich slices of the data, and it is applied to underrepresented, few-shot slices. We apply Ex2 to a range of language understanding tasks and significantly improve over state-of-the-art methods on multiple few-shot learning benchmarks, including for relation extraction (FewRel) and intent classification + slot filling (SNIPS).