mlsanityData to 7 Oct 2026

SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalization

arXiv 2601.22385 ↗First posted 29 Jan 2026cs.CL cs.AI cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) controls the trade-off between fitting preference labels and staying close to a reference model using a single global temperature beta, implicitly treating all preference pairs as equally informative. Real-world preference corpora are heterogeneous: they mix high-signal, objective failures (for example, safety, factuality, instruction violations) with low-signal or subjective distinctions (for example, style), and also include label noise. We introduce our method, SP2DPO (Semantic Per-Pair DPO), a generalization that replaces the global temperature with an instance-specific schedule beta_i pre-decided offline from structured semantic-gap annotations (category, magnitude, confidence) produced by teacher language models. We instantiate this procedure on the UltraFeedback preference corpus (59,960 pairs), enabling large-scale construction of an auditable beta_i artifact, and incur zero training-time overhead: the inner-loop optimizer remains standard DPO with beta set per pair. We focus our empirical study on AlpacaEval 2.0, reporting both raw win rate and length-controlled win rate. Across four open-weight, instruction-tuned student backbones (4B-8B), SP2DPO is competitive with a tuned global-beta DPO baseline and improves AlpacaEval 2.0 length-controlled win rate on two of four backbones, while avoiding per-model beta sweeps. All code, annotations, and artifacts will be released.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 48 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

32 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Dual-Difficulty Curriculum Learning for Direct Preference Optimization10 Apr
  2. 2025FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings11 Jan
  3. 2024Accelerated Preference Optimization for Large Language Model Alignment8 Oct
  4. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences12 Mar
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 24

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  12. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
12 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  2. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  4. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  5. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  6. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  7. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  8. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  9. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  10. 2024Curriculum Direct Preference Optimization for Diffusion and Consistency Modelscited by 36
  11. 2022Spurious Correlations in Reference-Free Evaluation of Text Generationcited by 26
  12. 2025CuDIP: Enhancing Theorem Proving in LLMs via Curriculum Learning-based Direct Preference Optimizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
  2. 2025DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization
  3. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
  4. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  5. 20242D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision
  6. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  7. 2026Small-Margin Preferences Still Matter-If You Train Them Right
  8. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts