mlsanityData to 7 Oct 2026

StalePO: Anchored Token-Level Preference Optimization using Legacy Post-Edits in Machine Translation

arXiv 2609.16340 ↗First posted 14 Sep 2026cs.CL cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Machine translation systems are periodically upgraded to stronger models, but the available preference signal is human post-edits of an older system's outputs, which the newer model may already surpass. Moreover, collecting fresh post-edits for every new model is prohibitively expensive. We call this the Stale Preference problem. Standard DPO can fail in this setting: it may increase the likelihood of inferior post-edits, erode the model's existing quality, and fail to provide the per-token control needed to correct localized errors. We introduce StalePO, an objective derived from three requirements this regime imposes. Likelihood movement must be downward on both responses, the policy must be anchored to its own base response, and the KL constraint must apply at the token level. These requirements are jointly necessary. In ablations, each mechanism in isolation leaves the model's performance indistinguishable from the base model, and only their combination converts stale feedback into gains. On English-to-Hindi and English-to-Turkish localization data, StalePO improves the fraction of segments passing all LLM-as-judge MQM quality checks by 14.9 and 4.6 percentage points, respectively, with gains concentrated on style and fluency. A human evaluation under the same framework confirms these gains on English-to-Hindi, raising the fraction of segments passing all seven human checks by 13.8 percentage points.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 31 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

23 of 31 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks7 Oct
  2. 2024Selective Preference Optimization via Token-Level Reward Function Estimation24 Aug
  3. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment12 Aug
  4. 2024BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization30 Jun
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  7. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  8. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  9. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards8 Feb
  10. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  11. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  12. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
3 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 8

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  5. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  6. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  7. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  8. 2024T-REG: Preference Optimization with Token-Level Reward Regularizationcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Post-edits Are Preferences Too
  2. 2025M2PO: Multi-Perspective Multi-Pair Preference Optimization for Machine Translation
  3. 2026PEGRL: Improving Machine Translation by Post-Editing Guided Reinforcement Learning
  4. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
  5. 2024Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation
  6. 2026CLewR: Curriculum Learning with Restarts for Machine Translation Preference Learning
  7. 2026TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
  8. 2025SimulPL: Aligning Human Preferences in Simultaneous Machine Translation