# CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translation

arXiv: https://arxiv.org/abs/2204.00665 · HTML: https://mlsanity.com/paper/2204.00665/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 66 references that are in the map.
- First posted on arXiv 1 Apr 2022.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](https://mlsanity.com/paper/1512.05287/) (2015)
- [The Curious Case of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2104.06683/) (2021)
- [A Survey on Multi-view Learning](https://mlsanity.com/paper/1304.5634/) (2013)
- [When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?](https://mlsanity.com/paper/1804.06323/) (2018)
- [Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View](https://mlsanity.com/paper/1906.02762/) (2019)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [Reward Augmented Maximum Likelihood for Neural Structured Prediction](https://mlsanity.com/paper/1609.00150/) (2016)
- [Data Noising as Smoothing in Neural Network Language Models](https://mlsanity.com/paper/1703.02573/) (2017)
- [Edinburgh Neural Machine Translation Systems for WMT 16](https://mlsanity.com/paper/1606.02891/) (2016)
- [On the Impact of Various Types of Noise on Neural Machine Translation](https://mlsanity.com/paper/1805.12282/) (2018)
- [SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation](https://mlsanity.com/paper/1808.07512/) (2018)
- [Sequence-Level Mixed Sample Data Augmentation](https://mlsanity.com/paper/2011.09039/) (2020)
- [Investigating Multilingual NMT Representations at Scale](https://mlsanity.com/paper/1909.02197/) (2019)
- [Generalized Data Augmentation for Low-Resource Translation](https://mlsanity.com/paper/1906.03785/) (2019)
- [Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation](https://mlsanity.com/paper/1902.01509/) (2019)
- [Soft Contextual Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/1905.10523/) (2019)

## Built on by (most cited first, 9 of 9)

- [Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation](https://mlsanity.com/paper/2404.14827/) (2024)
- [Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning](https://mlsanity.com/paper/2304.10309/) (2023)
- [Breaking the Representation Bottleneck of Chinese Characters: Neural Machine Translation with Stroke Sequence Modeling](https://mlsanity.com/paper/2211.12781/) (2022)
- [ConsistTL: Modeling Consistency in Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2212.04262/) (2022)
- [Importance-Aware Data Augmentation for Document-Level Neural Machine Translation](https://mlsanity.com/paper/2401.15360/) (2024)
- [Pluggable Neural Machine Translation Models via Memory-augmented Adapters](https://mlsanity.com/paper/2307.06029/) (2023)
- [Unraveling Key Factors of Knowledge Distillation](https://mlsanity.com/paper/2312.08585/) (2023)
- [Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMs](https://mlsanity.com/paper/2601.05635/) (2026)
- [SSDAU: Structured Semantic Data Augmentation for Joint Entity and Relation Extraction](https://mlsanity.com/paper/2605.23440/) (2026)

## Abstract (arXiv)

We propose a novel data-augmentation technique for neural machine translation based on ROT-k ciphertexts. ROT-k is a simple letter substitution cipher that replaces a letter in the plaintext with the kth letter after it in the alphabet. We first generate multiple ROT-k ciphertexts using different values of k for the plaintext which is the source side of the parallel data. We then leverage this enciphered training data along with the original parallel data via multi-source training to improve neural machine translation. Our method, CipherDAug, uses a co-regularization-inspired training procedure, requires no external data sources other than the original training data, and uses a standard Transformer to outperform strong data augmentation techniques on several datasets by a significant margin. This technique combines easily with existing approaches to data augmentation, and yields particularly strong results in low-resource settings.