mlsanityData to 7 Oct 2026

Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model

arXiv 2401.12873 ↗First posted 23 Jan 2024cs.CL cs.AIcited by 33

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Insufficient modeling of human preferences within the reward model is a major obstacle for leveraging human feedback to improve translation quality. Fortunately, quality estimation (QE), which predicts the quality of a given translation without reference, has achieved impressive alignment with human evaluations in the last two years. In this work, we investigate the potential of employing the QE model as the reward model to predict human preferences for feedback training. We first identify the overoptimization problem during QE-based feedback training, manifested as an increase in reward while translation quality declines. We examine the problem and argue that the vulnerability of the QE model might lead to high rewards for incorrect translations, resulting in overoptimization and error propagation. To address the problem, we adopt a simple yet effective method that uses heuristic rules to detect the incorrect translations and assigns a penalty term to the reward scores of them. Experimental results show that the proposed QE-based feedback training achieves consistent and significant improvements across various settings, further verified through human preference studies. Our subsequent analysis demonstrates the high data efficiency of the proposed QE-based feedback training: it outperforms systems using larger parallel corpora by a small amount of monolingual data. Our code is available at: https://github.com/zwhe99/FeedbackMT

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 59 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

40 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  2. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inference15 Nov
  3. 2023LIMA: Less Is More for Alignment18 May
  4. 2022Training language models to follow instructions with human feedback4 Mar
  5. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback24 Jul

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  12. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
23 more
  1. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  2. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  3. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  4. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  5. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  6. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  7. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  8. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  9. 2015Minimum Risk Training for Neural Machine Translationcited by 114
  10. 2019Beyond BLEU: Training Neural Machine Translation with Semantic Similaritycited by 84
  11. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  12. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  13. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 65
  14. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 58
  15. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  16. 2023MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methodscited by 33
  17. 2017Bandit Structured Prediction for Neural Sequence-to-Sequence Learningcited by 31
  18. 2023BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Trainingcited by 24
  19. 2017Nematus: a Toolkit for Neural Machine Translationcited by 16
  20. 2017Counterfactual Learning from Bandit Feedback under Deterministic Logging: A Case Study in Statistical Machine Translationcited by 13
  21. 2016Stochastic Structured Prediction under Bandit Feedbackcited by 12
  22. 2022Tencent AI Lab - Shanghai Jiao Tong University Low-Resource Translation System for the WMT22 Translation Taskcited by 8
  23. 2017A Shared Task on Bandit Learning for Machine Translationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 10; 2026: 5.

33 papers.

Sort:

By area:

  1. 2024Measuring Bargaining Abilities of LLMs: A Benchmark and A Buyer-Enhancement Methodcited by 44
  2. 2024TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinementcited by 37
  3. 2024Can Watermarks Survive Translation? On the Cross-lingual Consistency of Text Watermark for Large Language Modelscited by 27
  4. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  5. 2025New Trends for Modern Machine Translation with Large Reasoning Modelscited by 17
  6. 2024Can Automatic Metrics Assess High-Quality Translations?cited by 16
  7. 2024QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translationcited by 16
  8. 2024Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solutioncited by 13
  9. 2024Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Levelcited by 11
  10. 2024Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewardscited by 11
  11. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappingscited by 11
  12. 2024Beyond Correlation: Interpretable Evaluation of Machine Translation Metricscited by 10
21 more
  1. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  2. 2025Learning to Substitute Words with Model-based Score Rankingcited by 10
  3. 2025MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modelingcited by 9
  4. 2025Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voicecited by 9
  5. 2024Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translationcited by 8
  6. 2025SimulPL: Aligning Human Preferences in Simultaneous Machine Translationcited by 7
  7. 2025Redefining Machine Translation on Social Network Services with Large Language Modelscited by 6
  8. 2024CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedbackcited by 5
  9. 2025Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimationcited by 4
  10. 2025M2PO: Multi-Perspective Multi-Pair Preference Optimization for Machine Translationcited by 3
  11. 2026Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translationcited by 3
  12. 2026LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translationcited by 3
  13. 2024Multi-perspective Alignment for Increasing Naturalness in Neural Machine Translationcited by 1
  14. 2026STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translationcited by 1
  15. 2024A Probability--Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptorscited by 0
  16. 2024QE-EBM: Using Quality Estimators as Energy Loss for Machine Translationcited by 0
  17. 2024PMMT: Preference Alignment in Multilingual Machine Translation via LLM Distillationcited by 0
  18. 2025Balancing Global Quality and Pronoun-Specific Feedback for Context-Aware Machine Translationcited by 0
  19. 2025RL from Teacher-Model Refinement: Gradual Imitation Learning for Machine Translationcited by 0
  20. 2026Beyond Scalar Scores: Reinforcement Learning for Error-Aware Quality Estimation of Machine Translationcited by 0
  21. 2026Reinforcement Learning Elicits Contextual Learning of Unseen Language Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modeling
  2. 2023Human Feedback is not Gold Standard
  3. 2025Early-Exit and Instant Confidence Translation Quality Estimation
  4. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings
  5. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models
  6. 2024Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimization
  7. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
  8. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedback