# Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation

arXiv: https://arxiv.org/abs/2212.12662 · HTML: https://mlsanity.com/paper/2212.12662/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 24 Dec 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Data Augmentation for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1705.00440/) (2017)
- [Phrase-Based & Neural Unsupervised Machine Translation](https://mlsanity.com/paper/1804.07755/) (2018)
- [When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?](https://mlsanity.com/paper/1804.06323/) (2018)
- [SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation](https://mlsanity.com/paper/1808.07512/) (2018)
- [Revisiting Low-Resource Neural Machine Translation: A Case Study](https://mlsanity.com/paper/1905.11901/) (2019)
- [Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention](https://mlsanity.com/paper/1908.11365/) (2019)
- [Optimizing Deeper Transformers on Small Datasets](https://mlsanity.com/paper/2012.15355/) (2020)
- [Training Deeper Neural Machine Translation Models with Transparent Attention](https://mlsanity.com/paper/1808.07561/) (2018)
- [Shallow-to-Deep Training for Neural Machine Translation](https://mlsanity.com/paper/2010.03737/) (2020)
- [Pre-trained Language Model Representations for Language Generation](https://mlsanity.com/paper/1903.09722/) (2019)
- [Simple Fusion: Return of the Language Model](https://mlsanity.com/paper/1809.00125/) (2018)
- [On the use of BERT for Neural Machine Translation](https://mlsanity.com/paper/1909.12744/) (2019)
- [Learning Light-Weight Translation Models from Deep Transformer](https://mlsanity.com/paper/2012.13866/) (2020)
- [Improving Lexical Choice in Neural Machine Translation](https://mlsanity.com/paper/1710.01329/) (2017)
- [Depth Growing for Neural Machine Translation](https://mlsanity.com/paper/1907.01968/) (2019)
- [Multiscale Collaborative Deep Models for Neural Machine Translation](https://mlsanity.com/paper/2004.14021/) (2020)
- [Lipschitz Constrained Parameter Initialization for Deep Transformers](https://mlsanity.com/paper/1911.03179/) (2019)
- [Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Change](https://mlsanity.com/paper/2005.02008/) (2020)
- [Neutron: An Implementation of the Transformer Translation Model and its Variants](https://mlsanity.com/paper/1903.07402/) (2019)

## Abstract (arXiv)

In this paper, we study the use of deep Transformer translation model for the CCMT 2022 Chinese-Thai low-resource machine translation task. We first explore the experiment settings (including the number of BPE merge operations, dropout probability, embedding size, etc.) for the low-resource scenario with the 6-layer Transformer. Considering that increasing the number of layers also increases the regularization on new model parameters (dropout modules are also introduced when using more layers), we adopt the highest performance setting but increase the depth of the Transformer to 24 layers to obtain improved translation quality. Our work obtains the SOTA performance in the Chinese-to-Thai translation in the constrained evaluation.