# Few-shot learning through contextual data augmentation

arXiv: https://arxiv.org/abs/2103.16911 · HTML: https://mlsanity.com/paper/2103.16911/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations](https://mlsanity.com/paper/1805.06201/) (2018)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [Data Augmentation for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1705.00440/) (2017)
- [Conditional BERT Contextual Augmentation](https://mlsanity.com/paper/1812.06705/) (2018)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1808.08437/) (2018)
- [Soft Contextual Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/1905.10523/) (2019)
- [Regularization techniques for fine-tuning in neural machine translation](https://mlsanity.com/paper/1707.09920/) (2017)
- [Marian: Cost-effective High-Quality Neural Machine Translation in C++](https://mlsanity.com/paper/1805.12096/) (2018)
- [The University of Edinburgh's Submissions to the WMT19 News Translation Task](https://mlsanity.com/paper/1907.05854/) (2019)
- [One Sentence One Model for Neural Machine Translation](https://mlsanity.com/paper/1609.06490/) (2016)

## Built on by (most cited first, 8 of 8)

- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [Contrastive Learning for Prompt-Based Few-Shot Language Learners](https://mlsanity.com/paper/2205.01308/) (2022)
- [PAC-tuning:Fine-tuning Pretrained Language Models with PAC-driven Perturbed Gradient Descent](https://mlsanity.com/paper/2310.17588/) (2023)
- [Embedding Hallucination for Few-Shot Language Fine-tuning](https://mlsanity.com/paper/2205.01307/) (2022)
- [Improving Few-shot Generalization of Safety Classifiers via Data Augmented Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2310.16959/) (2023)
- [Generalization-Enhanced Few-Shot Object Detection in Remote Sensing](https://mlsanity.com/paper/2501.02474/) (2025)
- [Vicinal Risk Minimization for Few-Shot Cross-lingual Transfer in Abusive Language Detection](https://mlsanity.com/paper/2311.02025/) (2023)
- [A Comprehensive Survey of Challenges and Opportunities of Few-Shot Learning Across Multiple Domains](https://mlsanity.com/paper/2504.04017/) (2025)

## Abstract (arXiv)

Machine translation (MT) models used in industries with constantly changing topics, such as translation or news agencies, need to adapt to new data to maintain their performance over time. Our aim is to teach a pre-trained MT model to translate previously unseen words accurately, based on very few examples. We propose (i) an experimental setup allowing us to simulate novel vocabulary appearing in human-submitted translations, and (ii) corresponding evaluation metrics to compare our approaches. We extend a data augmentation approach using a pre-trained language model to create training examples with similar contexts for novel words. We compare different fine-tuning and data augmentation approaches and show that adaptation on the scale of one to five examples is possible. Combining data augmentation with randomly selected training sentences leads to the highest BLEU score and accuracy improvements. Impressively, with only 1 to 5 examples, our model reports better accuracy scores than a reference system trained with on average 313 parallel examples.