mlsanityData to 7 Oct 2026

Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimization

arXiv 2409.17673 ↗First posted 26 Sep 2024cs.CLcited by 5

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) and derivative techniques like Direct Preference Optimization (DPO) are task-alignment algorithms used to repurpose general, foundational models for specific tasks. We show that applying task-alignment to neural machine translation (NMT) addresses an existing task--data mismatch in NMT, leading to improvements across all languages of a multilingual model, even when task-alignment is only applied to a subset of those languages. We do so by introducing Direct Quality Optimization (DQO), a variant of DPO leveraging a pre-trained translation quality estimation model as a proxy for human preferences, and verify the improvements with both automatic metrics and human evaluation.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 74 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

32 of 74 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution18 Feb
  2. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  3. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inference15 Nov
  4. 2023Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding14 Nov

Other references · 28

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  10. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  11. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  12. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
16 more
  1. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  2. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  3. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205
  4. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  5. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  6. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  7. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  8. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  9. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  10. 2024A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelismcited by 33
  11. 2023There's no Data Like Better Data: Using QE Metrics for MT Data Filteringcited by 19
  12. 2015Building Subject-aligned Comparable Corpora and Mining it for Truly Parallel Sentence Pairscited by 15
  13. 2021XLEnt: Mining a Large Cross-lingual Entity Dataset with Lexical-Semantic-Phonetic Word Alignmentcited by 15
  14. 2023Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single Modelcited by 9
  15. 2019The Source-Target Domain Mismatch Problem in Machine Translationcited by 8
  16. 2024How to Learn in a Noisy World? Self-Correcting the Real-World Data Noise in Machine Translationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translationcited by 13
  2. 2026Cross-Preference Learning for Sentence-Level and Context-Aware Machine Translationcited by 2
  3. 2026Beyond Scalar Scores: Reinforcement Learning for Error-Aware Quality Estimation of Machine Translationcited by 0
  4. 2026Backtranslation Augmented Direct Preference Optimization for Neural Machine Translationcited by 0
  5. 2026LocQE: Principled Domain Adaptation for Localisation Quality Estimation by Leveraging Post-Editscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  2. 2026Meta-Learning Preferences for Multilingual LLM Alignment
  3. 2024Towards Efficient Exact Optimization of Language Model Alignment
  4. 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment
  5. 2024Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis
  6. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
  7. 2025A Principled Loss Function for Direct Language Model Alignment
  8. 2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment