PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
Direct Preference Optimization (DPO) simplifies alignment through pairwise comparisons but assumes all observed preferences are reliable. Real data often violates this assumption, leading to reversed, weak, or ambiguous labels that cause harmful policy updates. To address this, we propose Posterior Label Correction DPO (PLC-DPO) to robustly optimize preferences by routing each pair's training signal as a clean, flip, or tie case. The key idea is to use the calibrated policy-reference margin as online evidence to take appropriate correction actions. This reframes noisy preference learning as actively correcting supervision direction and strength rather than merely filtering suspicious examples. Across 57 dataset-model-benchmark cells, PLC-DPO obtains the best mean win rate against DPO (60.5 vs. 55.5 for the next-best method). Injected-noise and tie stress tests, human disagreement analysis, and self-confirmation diagnostics further show that the routing remains stable and distinguishes flipped from weakly directional pairs.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 36 references that are in the map. First posted on arXiv 31 Aug 2026.
Builds on
22 of 36 references.
Closest ancestors · 10
Same direction, newest first
- 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment29 Sep
- 2025Robust Preference Optimization via Dynamic Target Margins4 Jun
- 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples11 Feb
- 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
- 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
- 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
- 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
- 2023ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference5 Dec
- 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
- 2022Training language models to follow instructions with human feedback4 Mar
Other references · 12
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
- 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
- 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
- 20218-bit Optimizers via Block-wise Quantizationcited by 403
- 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
- 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Listwise Direct Preference Optimization with Multi-Dimensional Preference Mixing
- 2024Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization
- 2025Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
- 2026Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization
- 2025Why DPO is a Misspecified Estimator and How to Fix It
- 2025C2-DPO: Constrained Controlled Direct Preference Optimization
- 2026Mitigating Mismatch within Reference-based Preference Optimization
- 2026Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization