mlsanityData to 7 Oct 2026

Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation

arXiv 2401.08417 ↗First posted 16 Jan 2024cs.CLcited by 390

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Moderate-sized large language models (LLMs) -- those with 7B or 13B parameters -- exhibit promising machine translation (MT) performance. However, even the top-performing 13B LLM-based translation models, like ALMA, does not match the performance of state-of-the-art conventional encoder-decoder translation models or larger-scale LLMs such as GPT-4. In this study, we bridge this performance gap. We first assess the shortcomings of supervised fine-tuning for LLMs in the MT task, emphasizing the quality issues present in the reference data, despite being human-generated. Then, in contrast to SFT which mimics reference translations, we introduce Contrastive Preference Optimization (CPO), a novel approach that trains models to avoid generating adequate but not perfect translations. Applying CPO to ALMA models with only 22K parallel sentences and 12M parameters yields significant improvements. The resulting model, called ALMA-R, can match or exceed the performance of the WMT competition winners and GPT-4 on WMT'21, WMT'22 and WMT'23 test datasets.

Key facts. Cited by 390 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 47 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

31 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models19 Jun
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2023Mistral 7Bcited by 3,065
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
17 more
  1. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2020Contrastive Learning with Hard Negative Samplescited by 619
  5. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  6. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  7. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  8. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  9. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 88
  10. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  11. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  12. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  13. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 55
  14. 2023TIM: Teaching Large Language Models to Translate with Comparisoncited by 50
  15. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  16. 2023Improving Translation Faithfulness of Large Language Models via Augmenting Instructionscited by 28
  17. 2022Multilingual Representation Distillation with Contrastive Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 125; 2025: 182; 2026: 83.

390 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  3. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  4. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  5. 2024Iterative Reasoning Preference Optimizationcited by 244
  6. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  7. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  8. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  9. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  10. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  11. 2024Large Language Models Meet NLP: A Surveycited by 102
  12. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 102
38 more
  1. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  2. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  3. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  4. 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learningcited by 82
  5. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  6. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  7. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 64
  8. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  9. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  10. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  11. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  12. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learningcited by 51
  13. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  14. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  15. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  16. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  17. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  18. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  19. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMscited by 43
  20. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40
  21. 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMscited by 40
  22. 2025DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Drivingcited by 39
  23. 2025Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Studycited by 38
  24. 2024Model Extrapolation Expedites Alignmentcited by 37
  25. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  26. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
  27. 2025Understanding the Repeat Curse in Large Language Models from a Feature Perspectivecited by 36
  28. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Taskscited by 35
  29. 2025A Survey of Direct Preference Optimizationcited by 35
  30. 2024Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Modelcited by 33
  31. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languagescited by 31
  32. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  33. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29
  34. 2025Fine-tuning Large Language Models for Improving Factuality in Legal Question Answeringcited by 28
  35. 2024From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Modelscited by 27
  36. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 27
  37. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  38. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 26

50 most cited of 390 All 390 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Preference-driven Paradigm for Enhanced Translation with Large Language Models
  2. 2025Improving LLMs for Machine Translation Using Synthetic Preference Data
  3. 2025Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation
  4. 2024What do Large Language Models Need for Machine Translation Evaluation?
  5. 2024Refining Translations with LLMs: A Constraint-Aware Iterative Prompting Approach
  6. 2025Sequence-level Large Language Model Training with Contrastive Preference Optimization
  7. 2024Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languages
  8. 2024Large Language Model for Multi-Domain Translation: Benchmarking and Domain CoT Fine-tuning