mlsanityData to 7 Oct 2026

Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution

arXiv 2402.11525 ↗First posted 18 Feb 2024cs.CL cs.LGcited by 13

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Faithfulness, expressiveness, and elegance is the constant pursuit in machine translation. However, traditional metrics like textitBLEU do not strictly align with human preference of translation quality. In this paper, we explore leveraging reinforcement learning with human feedback (textitRLHF) to improve translation quality. It is non-trivial to collect a large high-quality dataset of human comparisons between translations, especially for low-resource languages. To address this issue, we propose a cost-effective preference learning strategy, optimizing reward models by distinguishing between human and machine translations. In this manner, the reward model learns the deficiencies of machine translation compared to human and guides subsequent improvements in machine translation. Experimental results demonstrate that textitRLHF can effectively enhance translation quality and this improvement benefits other translation directions not trained with textitRLHF. Further analysis indicates that the model's language capabilities play a crucial role in preference learning. A reward model with strong language capabilities can more sensitively learn the subtle differences in translation quality and align better with real human translation preferences.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 29 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

21 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model23 Jan
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 19

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  7. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  8. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  9. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  10. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205
  11. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  12. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
7 more
  1. 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
  2. 2018Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluationcited by 88
  3. 2019Beyond BLEU: Training Neural Machine Translation with Semantic Similaritycited by 84
  4. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  5. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  6. 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
  7. 2020A Survey of Deep Learning Techniques for Neural Machine Translationcited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024SpeechAlign: Aligning Speech Generation to Human Preferencescited by 45
  2. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 42
  3. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  4. 2025SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translationcited by 13
  5. 2025Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voicecited by 9
  6. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translationcited by 8
  7. 2025SimulPL: Aligning Human Preferences in Simultaneous Machine Translationcited by 7
  8. 2024Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimizationcited by 5
  9. 2024CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedbackcited by 5
  10. 2024Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysiscited by 4
  11. 2025CRPO: Confidence-Reward Driven Preference Optimization for Machine Translationcited by 3
  12. 2025Data-Efficient Domain Adaptation for LLM-based MT using Contrastive Preference Optimizationcited by 0
1 more
  1. 2026VEPO: Variable Entropy Policy Optimization for Low-Resource Language Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inference
  2. 2025RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation
  3. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  4. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  5. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings
  6. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  7. 2025RL from Teacher-Model Refinement: Gradual Imitation Learning for Machine Translation
  8. 2024How to Evaluate Reward Models for RLHF