# SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation

arXiv: https://arxiv.org/abs/1808.07512 · HTML: https://mlsanity.com/paper/1808.07512/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2018.

## Built on by (most cited first, 30 of 72)

- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better](https://mlsanity.com/paper/2106.08962/) (2021)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models](https://mlsanity.com/paper/2210.05497/) (2022)
- [Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learning](https://mlsanity.com/paper/2103.07552/) (2021)
- [Substructure Substitution: Structured Data Augmentation for NLP](https://mlsanity.com/paper/2101.00411/) (2021)
- [TreeMix: Compositional Constituency-based Data Augmentation for Natural Language Understanding](https://mlsanity.com/paper/2205.06153/) (2022)
- [Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Models](https://mlsanity.com/paper/2104.07578/) (2021)
- [PCC: Paraphrasing with Bottom-k Sampling and Cyclic Learning for Curriculum Data Augmentation](https://mlsanity.com/paper/2208.08110/) (2022)
- [Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecycle](https://mlsanity.com/paper/2303.01980/) (2023)
- [Phrase-level Active Learning for Neural Machine Translation](https://mlsanity.com/paper/2106.11375/) (2021)
- [Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.06812/) (2022)
- [To Augment or Not to Augment? A Comparative Study on Text Augmentation Techniques for Low-Resource NLP](https://mlsanity.com/paper/2111.09618/) (2021)
- [Data Augmentation by Concatenation for Low-Resource Translation: A Mystery and a Solution](https://mlsanity.com/paper/2105.01691/) (2021)
- [Grammar-Based Grounded Lexicon Learning](https://mlsanity.com/paper/2202.08806/) (2022)
- [Model-Agnostic Meta-Learning for Multilingual Hate Speech Detection](https://mlsanity.com/paper/2303.02513/) (2023)
- [Does DetectGPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Better](https://mlsanity.com/paper/2402.00263/) (2024)
- [Data Augmentation for Low-Resource Named Entity Recognition Using Backtranslation](https://mlsanity.com/paper/2108.11703/) (2021)
- [GDA: Generative Data Augmentation Techniques for Relation Extraction Tasks](https://mlsanity.com/paper/2305.16663/) (2023)
- [On-the-Fly Aligned Data Augmentation for Sequence-to-Sequence ASR](https://mlsanity.com/paper/2104.01393/) (2021)
- [CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.00665/) (2022)
- [Exploring Temporally Dynamic Data Augmentation for Video Recognition](https://mlsanity.com/paper/2206.15015/) (2022)
- [Text Augmentation in a Multi-Task View](https://mlsanity.com/paper/2101.05469/) (2021)
- [Sentence Concatenation Approach to Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2104.08478/) (2021)
- [Rethinking Data Augmentation for Low-Resource Neural Machine Translation: A Multi-Task Learning Approach](https://mlsanity.com/paper/2109.03645/) (2021)
- [Compositionality as Lexical Symmetry](https://mlsanity.com/paper/2201.12926/) (2022)
- [BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation](https://mlsanity.com/paper/2204.07837/) (2022)

## Abstract (arXiv)

In this work, we examine methods for data augmentation for text-based tasks such as neural machine translation (NMT). We formulate the design of a data augmentation policy with desirable properties as an optimization problem, and derive a generic analytic solution. This solution not only subsumes some existing augmentation schemes, but also leads to an extremely simple data augmentation strategy for NMT: randomly replacing words in both the source sentence and the target sentence with other random words from their corresponding vocabularies. We name this method SwitchOut. Experiments on three translation datasets of different scales show that SwitchOut yields consistent improvements of about 0.5 BLEU, achieving better or comparable performances to strong alternatives such as word dropout (Sennrich et al., 2016a). Code to implement this method is included in the appendix.