# Soft Contextual Data Augmentation for Neural Machine Translation

arXiv: https://arxiv.org/abs/1905.10523 · HTML: https://mlsanity.com/paper/1905.10523/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2019.

## Built on by (most cited first, 30 of 49)

- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://mlsanity.com/paper/2106.07499/) (2021)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [Prevent the Language Model from being Overconfident in Neural Machine Translation](https://mlsanity.com/paper/2105.11098/) (2021)
- [Gradient Imitation Reinforcement Learning for Low Resource Relation Extraction](https://mlsanity.com/paper/2109.06415/) (2021)
- [Substructure Substitution: Structured Data Augmentation for NLP](https://mlsanity.com/paper/2101.00411/) (2021)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Smoothed Contrastive Learning for Unsupervised Sentence Embedding](https://mlsanity.com/paper/2109.04321/) (2021)
- [Data Augmentation for Cross-Domain Named Entity Recognition](https://mlsanity.com/paper/2109.01758/) (2021)
- [Text Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks](https://mlsanity.com/paper/2202.13840/) (2022)
- [Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.06812/) (2022)
- [To Augment or Not to Augment? A Comparative Study on Text Augmentation Techniques for Low-Resource NLP](https://mlsanity.com/paper/2111.09618/) (2021)
- [UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost](https://mlsanity.com/paper/2104.04946/) (2021)
- [Data Augmentation by Concatenation for Low-Resource Translation: A Mystery and a Solution](https://mlsanity.com/paper/2105.01691/) (2021)
- [GDA: Generative Data Augmentation Techniques for Relation Extraction Tasks](https://mlsanity.com/paper/2305.16663/) (2023)
- [Towards Efficiently Diversifying Dialogue Generation via Embedding Augmentation](https://mlsanity.com/paper/2103.01534/) (2021)
- [CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.00665/) (2022)
- [Few-shot learning through contextual data augmentation](https://mlsanity.com/paper/2103.16911/) (2021)
- [Rethinking Data Augmentation for Low-Resource Neural Machine Translation: A Multi-Task Learning Approach](https://mlsanity.com/paper/2109.03645/) (2021)
- [BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation](https://mlsanity.com/paper/2204.07837/) (2022)
- [Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review](https://mlsanity.com/paper/2505.04531/) (2025)
- [EPiDA: An Easy Plug-in Data Augmentation Framework for High Performance Text Classification](https://mlsanity.com/paper/2204.11205/) (2022)
- [Neural Machine Translation for the Indigenous Languages of the Americas: An Introduction](https://mlsanity.com/paper/2306.06804/) (2023)
- [Aspect-Based Sentiment Analysis with Explicit Sentiment Augmentations](https://mlsanity.com/paper/2312.10961/) (2023)
- [Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation](https://mlsanity.com/paper/2203.02951/) (2022)
- [A Large Language Model and Denoising Diffusion Framework for Targeted Design of Microstructures with Commands in Natural Language](https://mlsanity.com/paper/2409.14473/) (2024)
- [Unsupervised Document Embedding via Contrastive Augmentation](https://mlsanity.com/paper/2103.14542/) (2021)
- [OCR Improves Machine Translation for Low-Resource Languages](https://mlsanity.com/paper/2202.13274/) (2022)

## Abstract (arXiv)

While data augmentation is an important trick to boost the accuracy of deep learning methods in computer vision tasks, its study in natural language tasks is still very limited. In this paper, we present a novel data augmentation method for neural machine translation. Different from previous augmentation methods that randomly drop, swap or replace words with other words in a sentence, we softly augment a randomly chosen word in a sentence by its contextual mixture of multiple related words. More accurately, we replace the one-hot representation of a word by a distribution (provided by a language model) over the vocabulary, i.e., replacing the embedding of this word by a weighted combination of multiple semantically similar words. Since the weights of those words depend on the contextual information of the word to be replaced, the newly generated sentences capture much richer information than previous augmentation methods. Experimental results on both small scale and large scale machine translation datasets demonstrate the superiority of our method over strong baselines.