mlsanityData to 14 Oct 2026

Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation

arXiv 2212.12662 ↗First posted 24 Dec 2022cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the use of deep Transformer translation model for the CCMT 2022 Chinese-Thai low-resource machine translation task. We first explore the experiment settings (including the number of BPE merge operations, dropout probability, embedding size, etc.) for the low-resource scenario with the 6-layer Transformer. Considering that increasing the number of layers also increases the regularization on new model parameters (dropout modules are also introduced when using more layers), we adopt the highest performance setting but increase the depth of the Transformer to 24 layers to obtain improved translation quality. Our work obtains the SOTA performance in the Chinese-to-Thai translation in the constrained evaluation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 24 Dec 2022.

Builds on

29 of 38 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2018Self-Attention with Relative Position Representationscited by 1,120
  7. 2020On Layer Normalization in the Transformer Architecturecited by 922
  8. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  9. 2017Six Challenges for Neural Machine Translationcited by 335
  10. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  11. 2017Data Augmentation for Low-Resource Neural Machine Translationcited by 139
  12. 2018Phrase-Based & Neural Unsupervised Machine Translationcited by 138
17 more
  1. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 126
  2. 2018SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translationcited by 72
  3. 2019Revisiting Low-Resource Neural Machine Translation: A Case Studycited by 61
  4. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
  5. 2020Optimizing Deeper Transformers on Small Datasetscited by 42
  6. 2018Training Deeper Neural Machine Translation Models with Transparent Attentioncited by 38
  7. 2020Shallow-to-Deep Training for Neural Machine Translationcited by 36
  8. 2019Pre-trained Language Model Representations for Language Generationcited by 34
  9. 2018Simple Fusion: Return of the Language Modelcited by 30
  10. 2019On the use of BERT for Neural Machine Translationcited by 23
  11. 2020Learning Light-Weight Translation Models from Deep Transformercited by 22
  12. 2017Improving Lexical Choice in Neural Machine Translationcited by 17
  13. 2019Depth Growing for Neural Machine Translationcited by 13
  14. 2020Multiscale Collaborative Deep Models for Neural Machine Translationcited by 11
  15. 2019Lipschitz Constrained Parameter Initialization for Deep Transformerscited by 9
  16. 2020Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Changecited by 4
  17. 2019Neutron: An Implementation of the Transformer Translation Model and its Variantscited by 1

Built on by

Not cited yet.