Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translation
Not on the map. Why?
Abstract (arXiv)
Zero-shot translation (ZST), which is generally based on a multilingual neural machine translation model, aims to translate between unseen language pairs in training data. The common practice to guide the zero-shot language mapping during inference is to deliberately insert the source and target language IDs, e.g., <EN> for English and <DE> for German. Recent studies have shown that language IDs sometimes fail to navigate the ZST task, making them suffer from the off-target problem (non-target language words exist in the generated translation) and, therefore, difficult to apply the current multilingual translation model to a broad range of zero-shot language scenarios. To understand when and why the navigation capabilities of language IDs are weakened, we compare two extreme decoder input cases in the ZST directions: Off-Target (OFF) and On-Target (ON) cases. By contrastively visualizing the contextual word representations (CWRs) of these cases with teacher forcing, we show that 1) the CWRs of different languages are effectively distributed in separate regions when the sentence and ID are matched (ON setting), and 2) if the sentence and ID are unmatched (OFF setting), the CWRs of different languages are chaotically distributed. Our analyses suggest that although they work well in ideal ON settings, language IDs become fragile and lose their navigation ability when faced with off-target tokens, which commonly exist during inference but are rare in training scenarios. In response, we employ unlikelihood tuning on the negative (OFF) samples to minimize their probability such that the language IDs can discriminate between the on- and off-target tokens during training. Experiments spanning 40 ZST directions show that our method reduces the off-target ratio by -48.0% on average, leading to a +9.1 BLEU improvement with only an extra +0.3% tuning cost.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 61 of its 73 references that are in the map. First posted on arXiv 28 Sep 2023.
Builds on
61 of 73 references.
References · 61
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
- 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
- 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
38 more
- 2016Bag of Tricks for Efficient Text Classificationcited by 876
- 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
- 2015Sequence Level Training with Recurrent Neural Networkscited by 576
- 2020Beyond English-Centric Multilingual Machine Translationcited by 561
- 2019Neural Text Generation with Unlikelihood Trainingcited by 503
- 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
- 2016FastText.zip: Compressing text classification modelscited by 335
- 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
- 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257
- 2019Massively Multilingual Neural Machine Translationcited by 246
- 2016An Actor-Critic Algorithm for Sequence Predictioncited by 159
- 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
- 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 126
- 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
- 2019Generalization in Generation: A closer look at Exposure Biascited by 123
- 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
- 2015Minimum Risk Training for Neural Machine Translationcited by 115
- 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
- 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
- 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 108
- 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
- 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
- 2021BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Datacited by 75
- 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
- 2019Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlationscited by 61
- 2019The Missing Ingredient in Zero-Shot Neural Machine Translationcited by 59
- 2020Understanding and Improving Lexical Choice in Non-Autoregressive Translationcited by 56
- 2016Zero-Resource Translation with Multi-Lingual Neural Machine Translationcited by 46
- 2019Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?cited by 35
- 2021Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translationcited by 34
- 2021Improving Multilingual Translation by Representation and Gradient Regularizationcited by 31
- 2018Multilingual Neural Machine Translation with Task-Specific Attentioncited by 26
- 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
- 2021Multilingual Unsupervised Neural Machine Translation with Denoising Adapterscited by 22
- 2020Beyond [CLS] through Ranking by Generationcited by 20
- 2022On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translationcited by 15
- 2021Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variablescited by 14
- 2021Alternative Input Signals Ease Transfer in Multilingual Machine Translationcited by 14
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2024: 2.
2 papers.