mlsanityData to 7 Oct 2026

Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation

arXiv 2305.09312 ↗First posted 16 May 2023cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

This paper studies the impact of layer normalization (LayerNorm) on zero-shot translation (ZST). Recent efforts for ZST often utilize the Transformer architecture as the backbone, with LayerNorm at the input of layers (PreNorm) set as the default. However, Xu et al. (2019) has revealed that PreNorm carries the risk of overfitting the training data. Based on this, we hypothesize that PreNorm may overfit supervised directions and thus have low generalizability for ZST. Through experiments on OPUS, IWSLT, and Europarl datasets for 54 ZST directions, we demonstrate that the original Transformer setting of LayerNorm after residual connections (PostNorm) consistently outperforms PreNorm by up to 12.3 BLEU points. We then study the performance disparities by analyzing the differences in off-target rates and structural variations between PreNorm and PostNorm. This study highlights the need for careful consideration of the LayerNorm setting for ZST.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

26 of 42 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Layer Normalizationcited by 5,579
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2019Root Mean Square Layer Normalizationcited by 1,425
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  9. 2017Mixed Precision Trainingcited by 1,135
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  11. 2020On Layer Normalization in the Transformer Architecturecited by 906
  12. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
14 more
  1. 2016FastText.zip: Compressing text classification modelscited by 331
  2. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  3. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  4. 2019Understanding and Improving Layer Normalizationcited by 213
  5. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  6. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  7. 2019Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlationscited by 61
  8. 2019The Missing Ingredient in Zero-Shot Neural Machine Translationcited by 59
  9. 2020Improving Zero-Shot Translation by Disentangling Positional Informationcited by 35
  10. 2019Improving Zero-shot Translation with Language-Independent Constraintscited by 33
  11. 2021Improving Multilingual Translation by Representation and Gradient Regularizationcited by 31
  12. 2021Language Tags Matter for Zero-Shot Neural Machine Translationcited by 29
  13. 2019Consistency by Agreement in Zero-shot Neural Machine Translationcited by 22
  14. 2021Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variablescited by 14

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2024Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languagescited by 3
  2. 2023Variable-length Neural Interlingua Representations for Zero-shot Neural Machine Translationcited by 0
  3. 2024How Far Can 100 Samples Go? Unlocking Overall Zero-Shot Multilingual Translation via Tiny Multi-Parallel Datacited by 0