mlsanityData to 13 Oct 2026

On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation

arXiv 2305.10930 ↗First posted 18 May 2023cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

While multilingual neural machine translation has achieved great success, it suffers from the off-target issue, where the translation is in the wrong language. This problem is more pronounced on zero-shot translation tasks. In this work, we find that failing in encoding discriminative target language signal will lead to off-target and a closer lexical distance (i.e., KL-divergence) between two languages' vocabularies is related with a higher off-target rate. We also find that solely isolating the vocab of different languages in the decoder can alleviate the problem. Motivated by the findings, we propose Language Aware Vocabulary Sharing (LAVS), a simple and effective algorithm to construct the multilingual vocabulary, that greatly alleviates the off-target problem of the translation model by increasing the KL-divergence between languages. We conduct experiments on a multilingual machine translation benchmark in 11 languages. Experiments show that the off-target rate for 90 translation tasks is reduced from 29% to 8%, while the overall BLEU score is improved by an average of 1.9 points without extra training cost or sacrificing the supervised directions' performance. We release the code at https://github.com/PKUnlp-icler/Off-Target-MNMT for reproduction.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Builds on 24 of its 28 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

24 of 28 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,106
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,480
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,916
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,910
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,979
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,487
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,473
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,302
  10. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  11. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  12. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
12 more
  1. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 254
  2. 2019Massively Multilingual Neural Machine Translationcited by 246
  3. 2016Toward Multilingual Neural Machine Translation with Universal Encoder and Decodercited by 76
  4. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 61
  5. 2019Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlationscited by 61
  6. 2020Vocabulary Learning via Optimal Transport for Neural Machine Translationcited by 58
  7. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  8. 2020Multi-task Learning for Multilingual Neural Machine Translationcited by 46
  9. 2021Improving Multilingual Translation by Representation and Gradient Regularizationcited by 31
  10. 2021Language Tags Matter for Zero-Shot Neural Machine Translationcited by 29
  11. 2020Subword Segmentation and a Single Bridge Language Affect Zero-Shot Neural Machine Translationcited by 13
  12. 2022Focus on the Target's Vocabulary: Masked Label Smoothing for Machine Translationcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1.

3 papers.

  1. 2023Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decodingcited by 10
  2. 2024Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuningcited by 3
  3. 2023On the Copying Problem of Unsupervised NMT: A Training Schedule with a Language Discriminator Losscited by 1