mlsanityData to 14 Oct 2026

Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation

arXiv 2410.07779 ↗First posted 10 Oct 2024cs.CLcited by 8

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Alignment with human preferences is an important step in developing accurate and safe large language models. This is no exception in machine translation (MT), where better handling of language nuances and context-specific variations leads to improved quality. However, preference data based on human feedback can be very expensive to obtain and curate at a large scale. Automatic metrics, on the other hand, can induce preferences, but they might not match human expectations perfectly. In this paper, we propose an approach that leverages the best of both worlds. We first collect sentence-level quality assessments from professional linguists on translations generated by multiple high-quality MT systems and evaluate the ability of current automatic metrics to recover these preferences. We then use this analysis to curate a new dataset, MT-Pref (metric induced translation preference) dataset, which comprises 18k instances covering 18 language directions, using texts sourced from multiple domains post-2022. We show that aligning TOWER models on MT-Pref significantly improves translation quality on WMT23 and FLORES benchmarks.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 54 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

39 of 54 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  2. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  3. 2024Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model23 Jan
  4. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  5. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  6. 2023Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding14 Nov
  7. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  8. 2022Training language models to follow instructions with human feedback4 Mar
  9. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback24 Jul

Other references · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  8. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  9. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  10. 2017Six Challenges for Neural Machine Translationcited by 335
  11. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  12. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
18 more
  1. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  2. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  3. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 145
  4. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 138
  5. 2015Minimum Risk Training for Neural Machine Translationcited by 115
  6. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  7. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  8. 2019Beyond BLEU: Training Neural Machine Translation with Semantic Similaritycited by 84
  9. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 72
  10. 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
  11. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 56
  12. 2023TIM: Teaching Large Language Models to Translate with Comparisoncited by 50
  13. 2023Steering Large Language Models for Machine Translation with Finetuning and In-Context Learningcited by 49
  14. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  15. 2018Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MTcited by 21
  16. 2024Fine-Tuned Machine Translation Metrics Struggle in Unseen Domainscited by 21
  17. 2020Correct Me If You Can: Learning from Error Corrections and Markingscited by 11
  18. 2024Did Translation Models Get More Robust Without Anyone Even Noticing?cited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 2.

8 papers.

  1. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappingscited by 11
  2. 2025MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modelingcited by 10
  3. 2025Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answeringcited by 10
  4. 2025M2PO: Multi-Perspective Multi-Pair Preference Optimization for Machine Translationcited by 4
  5. 2025Semi-Supervised Preference Optimization with Limited Feedbackcited by 3
  6. 2026Cross-Preference Learning for Sentence-Level and Context-Aware Machine Translationcited by 2
  7. 2026STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translationcited by 1
  8. 2025RL from Teacher-Model Refinement: Gradual Imitation Learning for Machine Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis
  2. 2024PMMT: Preference Alignment in Multilingual Machine Translation via LLM Distillation
  3. 2024Post-edits Are Preferences Too
  4. 2025SimulPL: Aligning Human Preferences in Simultaneous Machine Translation
  5. 2025Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modeling
  6. 2024A Survey on Human Preference Learning for Large Language Models
  7. 2023On Diversified Preferences of Large Language Model Alignment
  8. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models