mlsanityData to 14 Oct 2026

RL from Teacher-Model Refinement: Gradual Imitation Learning for Machine Translation

arXiv 2507.22219 ↗First posted 29 Jul 2025cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

Preference-learning methods for machine translation (MT), such as Direct Preference Optimization (DPO), have shown strong gains but typically rely on large, carefully curated preference triplets and often struggle to generalize beyond their tuning domains. We propose Reinforcement Learning from Teacher-Model Refinement (RLfR), which replaces static triplets with on-policy, actor-conditioned refinements produced by a frozen teacher. At each step, the actor samples candidate translations, the teacher performs a minimal local edit of each draft, and the actor is reinforced to close the gap using a composite reward that combines scaled negative edit distance for lexical and structural fidelity with COMET for semantic adequacy. This formulation yields a stable, model-aware learning signal without requiring explicit preference datasets. Experiments on FLORES-200 (English to German, Spanish, Chinese, Korean, and Japanese) show that RLfR consistently outperforms strong MT-SFT, DPO, and fixed-reference RL baselines, improving semantic quality and entity preservation, and also achieves superior performance under LLM-based judge evaluations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 30 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

23 of 30 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings8 Nov
  2. 2024Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation10 Oct
  3. 2024Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model23 Jan
  4. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  7. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  8. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  9. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  10. 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
  11. 2025Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Studycited by 38
  12. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
7 more
  1. 2024G-DIG: Towards Gradient-based Diverse and High-quality Instruction Data Selection for Machine Translationcited by 30
  2. 2024The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilitiescited by 22
  3. 2024Towards Cross-Cultural Machine Translation with Retrieval-Augmented Generation from Multilingual Knowledge Graphscited by 22
  4. 2025New Trends for Modern Machine Translation with Large Reasoning Modelscited by 17
  5. 2024MT-PATCHER: Selective and Extendable Knowledge Distillation from Large Language Models for Machine Translationcited by 10
  6. 2022MAD for Robust Reinforcement Learning in Machine Translationcited by 7
  7. 2025Mitigating Hallucinated Translations in Large Language Models with Hallucination-focused Preference Optimizationcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution
  2. 2026Backtranslation Augmented Direct Preference Optimization for Neural Machine Translation
  3. 2026Rewrite to Translate, Translate to Reward: Reinforcement Learning for Source Rewriting in Machine Translation
  4. 2025MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning
  5. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inference
  6. 2026CLewR: Curriculum Learning with Restarts for Machine Translation Preference Learning
  7. 2023Reinforced Self-Training (ReST) for Language Modeling
  8. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models