mlsanityData to 7 Oct 2026

A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models

arXiv 2309.11674 ↗First posted 20 Sep 2023cs.CLcited by 88

Not on the map. Why?

Abstract (arXiv)

Generative Large Language Models (LLMs) have achieved remarkable advancements in various NLP tasks. However, these advances have not been reflected in the translation task, especially those with moderate model sizes (i.e., 7B or 13B parameters), which still lag behind conventional supervised encoder-decoder translation models. Previous studies have attempted to improve the translation capabilities of these moderate LLMs, but their gains have been limited. In this study, we propose a novel fine-tuning approach for LLMs that is specifically designed for the translation task, eliminating the need for the abundant parallel data that traditional translation models usually depend on. Our approach consists of two fine-tuning stages: initial fine-tuning on monolingual data followed by subsequent fine-tuning on a small set of high-quality parallel data. We introduce the LLM developed through this strategy as Advanced Language Model-based trAnslator (ALMA). Based on LLaMA-2 as our underlying model, our results show that the model can achieve an average improvement of more than 12 BLEU and 12 COMET over its zero-shot performance across 10 translation directions from the WMT'21 (2 directions) and WMT'22 (8 directions) test datasets. The performance is significantly better than all prior work and even superior to the NLLB-54B model and GPT-3.5-text-davinci-003, with only 7B or 13B parameters. This method establishes the foundation for a novel training paradigm in machine translation.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Builds on 38 of its 59 references that are in the map. First posted on arXiv 20 Sep 2023.

Builds on

38 of 59 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
26 more
  1. 2023LIMA: Less Is More for Alignmentcited by 1,080
  2. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  4. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  5. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  6. 2023Textbooks Are All You Needcited by 561
  7. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  8. 2022UL2: Unifying Language Learning Paradigmscited by 292
  9. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  10. 2019Massively Multilingual Neural Machine Translationcited by 246
  11. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  12. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  13. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  14. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  15. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  16. 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
  17. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  18. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
  19. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 55
  20. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 55
  21. 2023TIM: Teaching Large Language Models to Translate with Comparisoncited by 50
  22. 2019The Effect of Translationese in Machine Translation Test Setscited by 49
  23. 2023BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Modelscited by 42
  24. 2021BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translationcited by 35
  25. 2023Improving Translation Faithfulness of Large Language Models via Augmenting Instructionscited by 28
  26. 2023Language-Aware Multilingual Machine Translation with Self-Supervised Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 37; 2025: 36; 2026: 11.

88 papers.

Sort:

By area:

  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  3. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  4. 2024Large Language Models Meet NLP: A Surveycited by 102
  5. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  6. 2024Scaling Laws for Downstream Task Performance of Large Language Modelscited by 49
  7. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  8. 2024Fine-tuning Large Language Models for Domain-specific Machine Translationcited by 43
  9. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
  10. 2024SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflectioncited by 34
  11. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  12. 2024Fine Tuning LLM for Enterprise: Practical Guidelines and Recommendationscited by 28
38 more
  1. 2025Defending against Indirect Prompt Injection by Instruction Detectioncited by 25
  2. 2024Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?cited by 23
  3. 2023Fine-tuning Large Language Models for Adaptive Machine Translationcited by 21
  4. 2025R1-T1: Fully Incentivizing Translation Capability in LLMs via Reasoning Learningcited by 21
  5. 2025Granary: Speech Recognition and Translation Dataset in 25 European Languagescited by 21
  6. 2025Seed-X: Building Strong Multilingual Translation LLM with 7B Parameterscited by 21
  7. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  8. 2024Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Modelscited by 17
  9. 2025SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignmentcited by 17
  10. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  11. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gatecited by 13
  12. 2025Compositional Translation: A Novel LLM-based Approach for Low-resource Machine Translationcited by 13
  13. 2024MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructionscited by 12
  14. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappingscited by 11
  15. 2025Multilingual Contextualization of Large Language Models for Document-Level Machine Translationcited by 11
  16. 2023Model-Based Minimum Bayes Risk Decoding for Text Generationcited by 10
  17. 2024Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Modelscited by 9
  18. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  19. 2025Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modelingcited by 9
  20. 2025See the Text: From Tokenization to Visual Readingcited by 9
  21. 2026From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Modelscited by 9
  22. 2025Science Across Languages: Assessing LLM Multilingual Translation of Scientific Paperscited by 8
  23. 2025Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translationcited by 8
  24. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translationcited by 8
  25. 2026The Finetuner's Fallacy: When to Pretrain with Your Finetuning Datacited by 7
  26. 2024How Multilingual Are Large Language Models Fine-Tuned for Translation?cited by 6
  27. 2024MageBench: Bridging Large Multimodal Models to Agentscited by 6
  28. 2025Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translationcited by 6
  29. 2025RaSA: Rank-Sharing Low-Rank Adaptationcited by 6
  30. 2024UCCIX: Irish-eXcellence Large Language Modelcited by 5
  31. 2024Boosting LLM Translation Skills without General Ability Loss via Rationale Distillationcited by 5
  32. 2024Direct Quantized Training of Language Models with Stochastic Roundingcited by 5
  33. 2024Make Imagination Clearer! Stable Diffusion-based Visual Imagination for Multimodal Machine Translationcited by 5
  34. 2025TACTIC: Translation Agents with Cognitive-Theoretic Interactive Collaborationcited by 5
  35. 2025Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMscited by 4
  36. 2025Improving Model Alignment Through Collective Intelligence of Open-Source LLMScited by 4
  37. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  38. 2024Language Modelling Approaches to Adaptive Machine Translationcited by 3

50 most cited of 88 All 88 (JSON)