# Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation

arXiv: https://arxiv.org/abs/2204.06812 · HTML: https://mlsanity.com/paper/2204.06812/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 74 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations](https://mlsanity.com/paper/1805.06201/) (2018)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [Dual Learning for Machine Translation](https://mlsanity.com/paper/1611.00179/) (2016)
- [Data Augmentation for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1705.00440/) (2017)
- [Phrase-Based & Neural Unsupervised Machine Translation](https://mlsanity.com/paper/1804.07755/) (2018)
- [Robust Neural Machine Translation with Doubly Adversarial Inputs](https://mlsanity.com/paper/1906.02443/) (2019)
- [Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks](https://mlsanity.com/paper/1909.00964/) (2019)
- [SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness](https://mlsanity.com/paper/2009.10195/) (2020)

## Built on by (most cited first, 13 of 13)

- [LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT](https://mlsanity.com/paper/2310.04673/) (2023)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation](https://mlsanity.com/paper/2209.03316/) (2022)
- [Alibaba-Translate China's Submission for WMT 2022 Metrics Shared Task](https://mlsanity.com/paper/2210.09683/) (2022)
- [LAE-ST-MoE: Boosted Language-Aware Encoder Using Speech Translation Auxiliary Task for E2E Code-switching ASR](https://mlsanity.com/paper/2309.16178/) (2023)
- [Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Models](https://mlsanity.com/paper/2306.08909/) (2023)
- [Causal Document-Grounded Dialogue Pre-training](https://mlsanity.com/paper/2305.10927/) (2023)
- [Alibaba-Translate China's Submission for WMT 2022 Quality Estimation Shared Task](https://mlsanity.com/paper/2210.10049/) (2022)
- [TreeSwap: Data Augmentation for Machine Translation via Dependency Subtree Swapping](https://mlsanity.com/paper/2311.02355/) (2023)
- [Data Augmentation for Machine Translation via Dependency Subtree Swapping](https://mlsanity.com/paper/2307.07025/) (2023)
- [EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learning](https://mlsanity.com/paper/2310.17233/) (2023)
- [Towards Reliable Neural Machine Translation with Consistency-Aware Meta-Learning](https://mlsanity.com/paper/2303.10966/) (2023)
- [Investigating Masking-based Data Generation in Language Models](https://mlsanity.com/paper/2307.00008/) (2023)

## Abstract (arXiv)

The principal task in supervised neural machine translation (NMT) is to learn to generate target sentences conditioned on the source inputs from a set of parallel sentence pairs, and thus produce a model capable of generalizing to unseen instances. However, it is commonly observed that the generalization performance of the model is highly influenced by the amount of parallel data used in training. Although data augmentation is widely used to enrich the training data, conventional methods with discrete manipulations fail to generate diverse and faithful training samples. In this paper, we present a novel data augmentation paradigm termed Continuous Semantic Augmentation (CsaNMT), which augments each training instance with an adjacency semantic region that could cover adequate variants of literal expression under the same meaning. We conduct extensive experiments on both rich-resource and low-resource settings involving various language pairs, including WMT14 English-{German,French}, NIST Chinese-English and multiple low-resource IWSLT translation tasks. The provided empirical evidences show that CsaNMT sets a new level of performance among existing augmentation techniques, improving on the state-of-the-art by a large margin. The core codes are contained in Appendix E.