Post-edits Are Preferences Too
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
Preference Optimization (PO) techniques are currently one of the state of the art techniques for fine-tuning large language models (LLMs) on pairwise preference feedback from human annotators. However, in machine translation, this sort of feedback can be difficult to solicit. Additionally, Kreutzer et al. (2018) have shown that, for machine translation, pairwise preferences are less reliable than other forms of human feedback, such as 5-point ratings. We examine post-edits to see if they can be a source of reliable human preferences by construction. In PO, a human annotator is shown sequences s1 and s2 and asked for a preference judgment, %s1 > s2; while for post-editing, editors create s1 and know that it should be better than s2. We attempt to use these implicit preferences for PO and show that it helps the model move towards post-edit-like hypotheses and away from machine translation-like hypotheses. Furthermore, we show that best results are obtained by pre-training the model with supervised fine-tuning (SFT) on post-edits in order to promote post-edit-like hypotheses to the top output ranks.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 26 references that are in the map. First posted on arXiv 3 Oct 2024.
Builds on
13 of 26 references.
Closest ancestors · 3
Same direction, newest first
- 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
- 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
- 2022Training language models to follow instructions with human feedback4 Mar
Other references · 10
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
- 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
- 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
- 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
- 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
- 2023Enhancing Supervised Learning with Contrastive Markings in Neural Machine Translation Trainingcited by 3
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.
- 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMscited by 40
- 2025Steering Large Language Models for Machine Translation Personalizationcited by 3
- 2026When Edit Localization Amplifies Relative Selection Bias: Gradient Geometry, Target Mismatch, and Importance Weightingcited by 0
- 2026When Edit Localization Amplifies Relative Selection Bias: Gradient Geometry, Target Mismatch, and Importance Weighting12 Sep 2026
- 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs20 Jun 2025
- 2025Steering Large Language Models for Machine Translation Personalization22 May 2025
Similar, not cited
Close in wording, no citation either way
- 2024Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation
- 2024Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis
- 2025M2PO: Multi-Perspective Multi-Pair Preference Optimization for Machine Translation
- 2024A Preference-driven Paradigm for Enhanced Translation with Large Language Models
- 2025Learning to Translate Ambiguous Terminology by Preference Optimization on Post-Edits
- 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization
- 2024Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution
- 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models