mlsanityData to 14 Oct 2026

Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languages

arXiv 2401.05811 ↗First posted 11 Jan 2024cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

This article introduces contrastive alignment instructions (AlignInstruct) to address two challenges in machine translation (MT) on large language models (LLMs). One is the expansion of supported languages to previously unseen ones. The second relates to the lack of data in low-resource languages. Model fine-tuning through MT instructions (MTInstruct) is a straightforward approach to the first challenge. However, MTInstruct is limited by weak cross-lingual signals inherent in the second challenge. AlignInstruct emphasizes cross-lingual supervision via a cross-lingual discriminator built using statistical word alignments. Our results based on fine-tuning the BLOOMZ models (1b1, 3b, and 7b1) in up to 24 unseen languages showed that: (1) LLMs can effectively translate unseen languages using MTInstruct; (2) AlignInstruct led to consistent improvements in translation quality across 48 translation directions involving English; (3) Discriminator-based instructions outperformed their generative counterparts as cross-lingual instructions; (4) AlignInstruct improved performance in 30 zero-shot directions.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 47 of its 78 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

47 of 78 references.

References · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
35 more
  1. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  2. 2017Mixed Precision Trainingcited by 1,156
  3. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  5. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  6. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  7. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  8. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  10. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  11. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  12. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
  13. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  14. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  15. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  16. 2022In-context Examples Selection for Machine Translationcited by 172
  17. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 145
  18. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  19. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  20. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  21. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  22. 2018Rapid Adaptation of Neural Machine Translation to New Languagescited by 82
  23. 2023Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translationcited by 74
  24. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  25. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  26. 2020Multi-task Learning for Multilingual Neural Machine Translationcited by 46
  27. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  28. 2020Improving Zero-Shot Translation by Disentangling Positional Informationcited by 35
  29. 2021Language Tags Matter for Zero-Shot Neural Machine Translationcited by 29
  30. 2022Cedille: A large autoregressive French language modelcited by 20
  31. 2019Explicit Cross-lingual Pre-training for Unsupervised Machine Translationcited by 13
  32. 2022Contrastive Demonstration Tuning for Pre-trained Language Modelscited by 7
  33. 2023Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translationcited by 3
  34. 2023Condensing Multilingual Knowledge with Lightweight Language-Specific Modulescited by 3
  35. 2022When do Contrastive Word Alignments Improve Many-to-many Neural Machine Translation?cited by 1

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  2. 2024Adapting Large Multimodal Models to Distribution Shifts: The Role of In-Context Learningcited by 4
  3. 2024Thinker-DDM: Modeling Deliberation for Machine Translation with a Drift-Diffusion Processcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Study
  2. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
  3. 2023Extrapolating Large Language Models to Non-English by Aligning Languages
  4. 2024Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages
  5. 2026Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment -- Is English Enough?
  6. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation
  7. 2023BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models
  8. 2023InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuning